Cold starts
Why the first Ostra Pro request after a quiet period can wait, how streaming and non-streaming requests behave while it starts, and how to warm it up ahead of time.
Ostra Decision is always on. Ostra Pro scales to zero after 5 minutes without requests, so nobody pays for idle capacity. The next request starts it again:
- a start usually takes under a minute;
- the first start after a model update can take up to about 5 minutes.
You're never charged for the wait or for a request that times out while it starts.
What your request sees
| Request | While Ostra Pro starts |
|---|---|
Streaming ("stream": true) | Answers 200 at once and waits for the model (up to 5 minutes), sending : ostra-pro is starting (Ns) comments every ~10 s. No retry needed. |
| Non-streaming | Waits up to 45 seconds. If the model still isn't up, fails with 503 service_unavailable, error.state: "starting" and Retry-After: 60. Nothing is charged. Retry after the delay. |
MCP ostra_pro_chat | Waits like a stream. Clients that accept text/event-stream get progress notifications (starting, then writing (about N tokens so far)). |
{
"error": {
"type": "service_unavailable",
"message": "Ostra Pro is starting up after a quiet period. Nothing was charged; retry shortly, or send stream: true to wait on the open connection.",
"retry_after_ms": 60000,
"state": "starting",
"request_id": "req_…"
}
}Avoiding the wait
Stream. Streaming requests absorb the start for you.
Warm up first. Before a burst (a CI job, an agent session), call POST /warm. It's free, works with any key and isn't rate limited. It answers 202 {"state": "starting", "retry_after_seconds": 30} while starting and 200 {"state": "ready"} once a replica is up.
# Wait until Ostra Pro is ready (up to 5 minutes), then continue.
for i in $(seq 1 10); do
state=$(curl -s -X POST https://ostra.testrelic.ai/api/v1/warm \
-H "Authorization: Bearer $OSTRA_API_KEY" | python3 -c 'import json,sys; print(json.load(sys.stdin)["state"])')
[ "$state" = "ready" ] && break
sleep 30
doneKeep it warm. Ostra Pro stays up while it gets at least one request every 5 minutes.
In the Console, the Playground shows Ostra Pro's state (Ready, Starting, Asleep) and has a Warm up now button.
Retrying non-streaming requests
OpenAI and Anthropic SDKs retry 503 automatically with a short backoff; give them enough attempts, or a longer timeout, for a start:
from openai import OpenAI
import os
client = OpenAI(
base_url="https://ostra.testrelic.ai/api/v1",
api_key=os.environ["OSTRA_API_KEY"],
max_retries=6, # default is 2
timeout=600,
)Or honour Retry-After yourself: see Errors → Retrying.