Streaming

Stream Ostra Pro answers as server-sent events — the OpenAI and Anthropic event formats, usage chunks, keep-alive comments during cold starts, and errors inside a stream.

Set "stream": true on /chat/completions or /messages to receive the answer as it's written. Streams answer 200 with content-type: text/event-stream straight away, before the model has started, and stay open while it works (up to 14 minutes).

Ostra Decision answers in milliseconds and doesn't stream.

OpenAI format (/chat/completions)

Each event is a data: line with a chat.completion.chunk; the stream ends with data: [DONE].

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","model":"ostra-pro","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","model":"ostra-pro","choices":[{"index":0,"delta":{"content":"CI machines are slower"},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","model":"ostra-pro","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]
  • With thinking on, reasoning arrives first in delta.reasoning_content, then the answer in delta.content.
  • Tool calls arrive in delta.tool_calls, as in OpenAI's API.
  • Add "stream_options": {"include_usage": true} to get one more chunk before [DONE] with choices: [] and the usage you were billed for.
from openai import OpenAI
import os

client = OpenAI(base_url="https://ostra.testrelic.ai/api/v1", api_key=os.environ["OSTRA_API_KEY"])
stream = client.chat.completions.create(
    model="ostra-pro",
    messages=[{"role": "user", "content": "Explain this stack trace: ..."}],
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        print("\n", chunk.usage)

Anthropic format (/messages)

Anthropic's events, unchanged: message_start, content_block_start, content_block_delta, content_block_stop, message_delta (with the final usage and stop_reason), message_stop, plus ping. There's no [DONE]. Anthropic SDKs' stream() helpers work as-is.

Keep-alive comments while Ostra Pro starts

If Ostra Pro is asleep, the stream waits for it to start (up to 5 minutes) and sends an SSE comment about every 10 seconds so proxies don't close the connection:

: ostra-pro is starting (10s)

: ostra-pro is starting (20s)

Comment lines start with : and every SSE parser ignores them. This is why streaming is the simplest way to call Ostra Pro after a quiet period: no 503s, no retry loop. See Cold starts.

Errors inside a stream

Errors found before the stream starts (validation, auth, balance, rate limit) are normal HTTP errors. Once it has started, a failure is sent as the last event, and the HTTP status stays 200:

data: {"error":{"type":"upstream_error","message":"The Ostra Pro stream broke before it finished. You were charged only for the tokens produced.","request_id":"req_…"}}

On /messages it's an Anthropic event: error. Treat a stream that ends without [DONE] (OpenAI format) or message_stop (Anthropic format) as failed.

Stopping a stream

Close the connection to stop generating. You're charged only for the tokens produced up to that point.

Was this page helpful?

On this page