Sessions and caching
Make follow-up Ostra Pro turns cheaper and faster with the x-ostra-session header, and how the Ostra Decision 24-hour answer cache works.
Ostra Pro: prompt caching and sessions
Ostra Pro keeps a prompt cache on each model replica. When a request starts with the same text as an earlier one (the system prompt, tool definitions and earlier turns of a conversation), that prefix is read from the cache instead of being processed again:
- it's billed as cached input at $0.06 per 1M tokens instead of $0.64, and
- the first token arrives sooner.
You'll see it in usage.prompt_tokens_details.cached_tokens (OpenAI format) or usage.cache_read_input_tokens (Anthropic format).
Ostra Pro can run on more than one replica, and each has its own cache. To keep a conversation on the replica that already holds its prefix, send x-ostra-session with one stable value per conversation:
curl https://ostra.testrelic.ai/api/v1/chat/completions \
-H "Authorization: Bearer $OSTRA_API_KEY" \
-H "Content-Type: application/json" \
-H "x-ostra-session: checkout-suite-run-42" \
-d '{"model": "ostra-pro", "messages": [ ... ]}'| Value | 1–128 characters: letters, digits, _, ., : or -. Invalid values are ignored. |
| Scope | One value per conversation or agent run. Reuse it for every turn. |
| Privacy | The value is scoped to your workspace; two workspaces using the same value never share a cache or a route. |
| Guarantee | Best effort. When replicas scale up or down, a session can move and lose its cache; the request still succeeds, it just bills more of its input at the normal rate. |
| Applies to | /chat/completions, /messages, and the MCP tool ostra_pro_chat (its session argument). |
from openai import OpenAI
import os
client = OpenAI(
base_url="https://ostra.testrelic.ai/api/v1",
api_key=os.environ["OSTRA_API_KEY"],
default_headers={"x-ostra-session": "checkout-suite-run-42"}, # one client per conversation
)Getting the most from the cache
- Keep the start of the prompt stable: system prompt first, then tools, then the conversation. Put anything that changes per request (timestamps, run ids) at the end.
- Send the whole conversation each turn, unchanged; only append.
- Use a new session value for an unrelated conversation; reusing one for everything spreads nothing and helps nothing.
Ostra Decision: the 24-hour answer cache
Ostra Decision caches answers, not prompts. A request with the same task and the same input as one in the last 24 hours, in the same workspace, returns the stored answer:
cached: true,billed.units: 0, cost $0;- the same
decision_idas the original answer; - it still counts toward your rate limit.
Inputs are compared after normalising JSON key order, so {"a":1,"b":2} and {"b":2,"a":1} match. Any other difference (one more history run, a changed error message) is a new request. The cache is cleared when a new Ostra Decision release goes live, so answers always come from the current model.
Streaming
Stream Ostra Pro answers as server-sent events — the OpenAI and Anthropic event formats, usage chunks, keep-alive comments during cold starts, and errors inside a stream.
Cold starts
Why the first Ostra Pro request after a quiet period can wait, how streaming and non-streaming requests behave while it starts, and how to warm it up ahead of time.