Cold starts

Why the first Ostra Pro request after a quiet period can wait, how streaming and non-streaming requests behave while it starts, and how to warm it up ahead of time.

Ostra Decision is always on. Ostra Pro scales to zero after 5 minutes without requests, so nobody pays for idle capacity. The next request starts it again:

  • a start usually takes under a minute;
  • the first start after a model update can take up to about 5 minutes.

You're never charged for the wait or for a request that times out while it starts.

What your request sees

RequestWhile Ostra Pro starts
Streaming ("stream": true)Answers 200 at once and waits for the model (up to 5 minutes), sending : ostra-pro is starting (Ns) comments every ~10 s. No retry needed.
Non-streamingWaits up to 45 seconds. If the model still isn't up, fails with 503 service_unavailable, error.state: "starting" and Retry-After: 60. Nothing is charged. Retry after the delay.
MCP ostra_pro_chatWaits like a stream. Clients that accept text/event-stream get progress notifications (starting, then writing (about N tokens so far)).
{
  "error": {
    "type": "service_unavailable",
    "message": "Ostra Pro is starting up after a quiet period. Nothing was charged; retry shortly, or send stream: true to wait on the open connection.",
    "retry_after_ms": 60000,
    "state": "starting",
    "request_id": "req_…"
  }
}

Avoiding the wait

Stream. Streaming requests absorb the start for you.

Warm up first. Before a burst (a CI job, an agent session), call POST /warm. It's free, works with any key and isn't rate limited. It answers 202 {"state": "starting", "retry_after_seconds": 30} while starting and 200 {"state": "ready"} once a replica is up.

# Wait until Ostra Pro is ready (up to 5 minutes), then continue.
for i in $(seq 1 10); do
  state=$(curl -s -X POST https://ostra.testrelic.ai/api/v1/warm \
    -H "Authorization: Bearer $OSTRA_API_KEY" | python3 -c 'import json,sys; print(json.load(sys.stdin)["state"])')
  [ "$state" = "ready" ] && break
  sleep 30
done

Keep it warm. Ostra Pro stays up while it gets at least one request every 5 minutes.

In the Console, the Playground shows Ostra Pro's state (Ready, Starting, Asleep) and has a Warm up now button.

Retrying non-streaming requests

OpenAI and Anthropic SDKs retry 503 automatically with a short backoff; give them enough attempts, or a longer timeout, for a start:

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://ostra.testrelic.ai/api/v1",
    api_key=os.environ["OSTRA_API_KEY"],
    max_retries=6,   # default is 2
    timeout=600,
)

Or honour Retry-After yourself: see Errors → Retrying.

Was this page helpful?

On this page