Streaming
Stream Ostra Pro answers as server-sent events — the OpenAI and Anthropic event formats, usage chunks, keep-alive comments during cold starts, and errors inside a stream.
Set "stream": true on /chat/completions or /messages to receive the answer as it's written. Streams answer 200 with content-type: text/event-stream straight away, before the model has started, and stay open while it works (up to 14 minutes).
Ostra Decision answers in milliseconds and doesn't stream.
OpenAI format (/chat/completions)
Each event is a data: line with a chat.completion.chunk; the stream ends with data: [DONE].
data: {"id":"chatcmpl-…","object":"chat.completion.chunk","model":"ostra-pro","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}
data: {"id":"chatcmpl-…","object":"chat.completion.chunk","model":"ostra-pro","choices":[{"index":0,"delta":{"content":"CI machines are slower"},"finish_reason":null}]}
data: {"id":"chatcmpl-…","object":"chat.completion.chunk","model":"ostra-pro","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]- With thinking on, reasoning arrives first in
delta.reasoning_content, then the answer indelta.content. - Tool calls arrive in
delta.tool_calls, as in OpenAI's API. - Add
"stream_options": {"include_usage": true}to get one more chunk before[DONE]withchoices: []and theusageyou were billed for.
from openai import OpenAI
import os
client = OpenAI(base_url="https://ostra.testrelic.ai/api/v1", api_key=os.environ["OSTRA_API_KEY"])
stream = client.chat.completions.create(
model="ostra-pro",
messages=[{"role": "user", "content": "Explain this stack trace: ..."}],
stream=True,
stream_options={"include_usage": True},
)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
if chunk.usage:
print("\n", chunk.usage)Anthropic format (/messages)
Anthropic's events, unchanged: message_start, content_block_start, content_block_delta, content_block_stop, message_delta (with the final usage and stop_reason), message_stop, plus ping. There's no [DONE]. Anthropic SDKs' stream() helpers work as-is.
Keep-alive comments while Ostra Pro starts
If Ostra Pro is asleep, the stream waits for it to start (up to 5 minutes) and sends an SSE comment about every 10 seconds so proxies don't close the connection:
: ostra-pro is starting (10s)
: ostra-pro is starting (20s)Comment lines start with : and every SSE parser ignores them. This is why streaming is the simplest way to call Ostra Pro after a quiet period: no 503s, no retry loop. See Cold starts.
Errors inside a stream
Errors found before the stream starts (validation, auth, balance, rate limit) are normal HTTP errors. Once it has started, a failure is sent as the last event, and the HTTP status stays 200:
data: {"error":{"type":"upstream_error","message":"The Ostra Pro stream broke before it finished. You were charged only for the tokens produced.","request_id":"req_…"}}On /messages it's an Anthropic event: error. Treat a stream that ends without [DONE] (OpenAI format) or message_stop (Anthropic format) as failed.
Stopping a stream
Close the connection to stop generating. You're charged only for the tokens produced up to that point.
Billing and usage
How every Ostra request is priced and charged — Ostra Decision per-unit prices, the 24-hour cache, batch discount, Ostra Pro token pricing and up-front holds, spend limits, and the cost headers.
Sessions and caching
Make follow-up Ostra Pro turns cheaper and faster with the x-ostra-session header, and how the Ostra Decision 24-hour answer cache works.