Sessions and caching

Make follow-up Ostra Pro turns cheaper and faster with the x-ostra-session header, and how the Ostra Decision 24-hour answer cache works.

Ostra Pro: prompt caching and sessions

Ostra Pro keeps a prompt cache on each model replica. When a request starts with the same text as an earlier one (the system prompt, tool definitions and earlier turns of a conversation), that prefix is read from the cache instead of being processed again:

  • it's billed as cached input at $0.06 per 1M tokens instead of $0.64, and
  • the first token arrives sooner.

You'll see it in usage.prompt_tokens_details.cached_tokens (OpenAI format) or usage.cache_read_input_tokens (Anthropic format).

Ostra Pro can run on more than one replica, and each has its own cache. To keep a conversation on the replica that already holds its prefix, send x-ostra-session with one stable value per conversation:

curl https://ostra.testrelic.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OSTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-ostra-session: checkout-suite-run-42" \
  -d '{"model": "ostra-pro", "messages": [ ... ]}'
Value1–128 characters: letters, digits, _, ., : or -. Invalid values are ignored.
ScopeOne value per conversation or agent run. Reuse it for every turn.
PrivacyThe value is scoped to your workspace; two workspaces using the same value never share a cache or a route.
GuaranteeBest effort. When replicas scale up or down, a session can move and lose its cache; the request still succeeds, it just bills more of its input at the normal rate.
Applies to/chat/completions, /messages, and the MCP tool ostra_pro_chat (its session argument).
from openai import OpenAI
import os

client = OpenAI(
    base_url="https://ostra.testrelic.ai/api/v1",
    api_key=os.environ["OSTRA_API_KEY"],
    default_headers={"x-ostra-session": "checkout-suite-run-42"},  # one client per conversation
)

Getting the most from the cache

  • Keep the start of the prompt stable: system prompt first, then tools, then the conversation. Put anything that changes per request (timestamps, run ids) at the end.
  • Send the whole conversation each turn, unchanged; only append.
  • Use a new session value for an unrelated conversation; reusing one for everything spreads nothing and helps nothing.

Ostra Decision: the 24-hour answer cache

Ostra Decision caches answers, not prompts. A request with the same task and the same input as one in the last 24 hours, in the same workspace, returns the stored answer:

  • cached: true, billed.units: 0, cost $0;
  • the same decision_id as the original answer;
  • it still counts toward your rate limit.

Inputs are compared after normalising JSON key order, so {"a":1,"b":2} and {"b":2,"a":1} match. Any other difference (one more history run, a changed error message) is a new request. The cache is cleared when a new Ostra Decision release goes live, so answers always come from the current model.

Was this page helpful?

On this page