Ostra Decision
One fast model with three CI tasks — failure triage, test impact and selector survival. How it decides, when it defers, what it needs from you, and its limits.
Ostra Decision (API id ostra-lite, Beta) answers the repetitive questions a CI pipeline asks thousands of times a day. It returns a class and a probability in milliseconds, at a price that makes it reasonable to call on every failure, diff and release.
| Task | Question | Billed per | Per 1M |
|---|---|---|---|
triage | Why did this test fail? | Failure | $0.50 |
test_impact | Which tests does this change affect? | Request | $2.00 |
selector_survival | Will this locator survive the next release? | Locator | $0.25 |
You pick the task per request: POST /api/v1/predict with {"model": "ostra-lite", "task": "...", "input": {...}}, or one MCP tool per task. The output it returns is free.
Evidence, not guesses
Ostra Decision decides from the evidence you send: the error, the code change, the test's recent history, what the DOM looked like. Three principles follow:
- Every evidence field is optional. Send what you have; leave out what you don't.
- Missing is not zero. A field you leave out is treated as unknown, never as
0orfalse. Don't fill gaps with zeros:flake_rate_50: 0says "this test never flakes", which is a strong claim. - No evidence, no answer. A request with nothing about this test (triage: no history, diff or DOM check; selector survival: no history and no DOM check) gets
deferwithdefer_reason: "insufficient_evidence". An error message on its own isn't enough to call a test flaky.
The more you send, the more often you get a firm answer.
Reading an answer
Every decision has the same core fields:
{
"decision_id": "dec_5b0c9e2a7f3d4e18a6b2c9d0e1f4a7b3",
"task": "triage",
"decision": "flaky",
"value": "flaky",
"p": 0.83,
"probabilities": { "flaky": 0.83, "product_bug": 0.06, "test_bug": 0.05, "environment": 0.04, "selector_drift": 0.02 },
"threshold": 0.6,
"defer_reason": null,
"decided_by": "model",
"rule": null,
"model": { "release": "v.beta.5", "version": "1.4.0", "content_hash": "sha256:3f9a1c07d2e4b8a6", "gate_passed": true }
}| Field | Meaning |
|---|---|
decision | The answer. Act on this. |
value | The same answer, or null when it's defer. Handy when you store answers. |
p | Probability of the top class. |
probabilities | Every class's probability. Useful to see the runner-up. |
threshold | The bar p had to clear for a firm answer. |
defer_reason | Why it deferred, when it did. |
decided_by, rule | rule when a certain environment rule answered (see below). |
model | The release and model version that answered. Log it. |
Confidence and defer
Each task has a threshold (reported in threshold; a class can carry a stricter or looser bar than the task default). When the top class doesn't clear it, the answer is defer rather than a weak guess. defer_reason says why:
defer_reason | Meaning | What to do |
|---|---|---|
below_threshold | The evidence points two ways. | Send it to a person, or add evidence (history, diff, DOM check) and ask again. |
insufficient_evidence | The request said nothing about this test or locator. | Add at least one piece of history, diff or DOM evidence. |
model_gate_failed | This task's model didn't pass its release quality gate, so it's answering defer on purpose. | Treat as "no answer" for now; it clears with the next release. |
A defer answer is billed like any other: it's a real answer ("not sure").
Likely flaky
Run history alone can't always tell a flaky test from an intermittent product bug, so triage has a hedged answer, likely_flaky: the test is probably flaky but the model isn't certain enough to say so. Quarantine and re-run it, but don't close the issue. In the current release, likely_flaky is only given when the request shows the test has passed on a retry at least once (memory.passed_on_retry_50 of 1 or more); otherwise the same evidence defers.
Rule-decided answers
Some failures are environment problems whatever else the context says. When the triage error_message matches one of these, a rule answers environment with p 0.99, decided_by: "rule" and the rule's id:
rule | Matches |
|---|---|
browser_not_installed | "Executable doesn't exist", "browsers are not installed", "please run the following command to download new browsers" |
host_missing_dependencies | "Host system is missing dependencies" |
webdriver_session_not_created | "SessionNotCreated", "session not created" |
port_in_use | EADDRINUSE |
disk_full | ENOSPC, "no space left on device" |
probabilities still holds the model's own view.
Releases and quality gates
Ostra Decision improves through releases (v.beta.5, for example). Each task's model must pass a quality gate before release; model.gate_passed is true in normal operation. A release can change answers for the same input but never the request or response shape. The live release is in GET /api/v1/models and in the Console under Models → Ostra Decision.
Caching
The same task with the same input, in the same workspace, within 24 hours, is answered from cache: cached: true, no charge. Key order in the JSON doesn't matter; any other difference makes a new request. A new release clears the cache. Cached requests still count toward your rate limit.
Batching
POST /api/v1/batch runs up to 100 tasks in one request, mixed tasks allowed, at half price. Use it whenever you have more than one thing to ask: all the failures of a CI run, all the locators of a page object. Each item answers on its own; one bad item doesn't fail the batch.
Limits
| Limit | |
|---|---|
| Request body | 2 MB |
| Items per batch | 100 |
| Candidates per test-impact request | 2,000 |
| Locators per selector-survival request | 500 |
| History entries per locator | 100 |
error_message | 8,000 characters |
Identifier fields (test_id, run_id, locator_id, …) | 1–512 characters |
| Answer time | Usually milliseconds. A request the model hasn't answered in 10 seconds fails with 504 upstream_timeout and isn't charged. |
Requests per minute depend on your workspace's usage level.
Where to use it
- REST:
POST /predict,POST /batch. - MCP:
ostra_triage_failure,ostra_select_impacted_tests,ostra_check_selector. - Console: Playground → Ostra Decision, with examples for every task.
Each answer has a decision_id. Log it with the test so you can match answers to outcomes later.
Quickstart
Sign in to the TestRelic Console, add credit, create an API key, try both models in the Playground, make your first API call and connect your coding agent over MCP.
Failure triage
Classify a failing test as flaky, likely flaky, product bug, test bug, environment or selector drift — what to send from Playwright, Cypress, Jest or Selenium, what each answer means and what to do next.