Decision

Ostra Decision

One fast model with three CI tasks — failure triage, test impact and selector survival. How it decides, when it defers, what it needs from you, and its limits.

Ostra Decision (API id ostra-lite, Beta) answers the repetitive questions a CI pipeline asks thousands of times a day. It returns a class and a probability in milliseconds, at a price that makes it reasonable to call on every failure, diff and release.

TaskQuestionBilled perPer 1M
triageWhy did this test fail?Failure$0.50
test_impactWhich tests does this change affect?Request$2.00
selector_survivalWill this locator survive the next release?Locator$0.25

You pick the task per request: POST /api/v1/predict with {"model": "ostra-lite", "task": "...", "input": {...}}, or one MCP tool per task. The output it returns is free.

Evidence, not guesses

Ostra Decision decides from the evidence you send: the error, the code change, the test's recent history, what the DOM looked like. Three principles follow:

  1. Every evidence field is optional. Send what you have; leave out what you don't.
  2. Missing is not zero. A field you leave out is treated as unknown, never as 0 or false. Don't fill gaps with zeros: flake_rate_50: 0 says "this test never flakes", which is a strong claim.
  3. No evidence, no answer. A request with nothing about this test (triage: no history, diff or DOM check; selector survival: no history and no DOM check) gets defer with defer_reason: "insufficient_evidence". An error message on its own isn't enough to call a test flaky.

The more you send, the more often you get a firm answer.

Reading an answer

Every decision has the same core fields:

{
  "decision_id": "dec_5b0c9e2a7f3d4e18a6b2c9d0e1f4a7b3",
  "task": "triage",
  "decision": "flaky",
  "value": "flaky",
  "p": 0.83,
  "probabilities": { "flaky": 0.83, "product_bug": 0.06, "test_bug": 0.05, "environment": 0.04, "selector_drift": 0.02 },
  "threshold": 0.6,
  "defer_reason": null,
  "decided_by": "model",
  "rule": null,
  "model": { "release": "v.beta.5", "version": "1.4.0", "content_hash": "sha256:3f9a1c07d2e4b8a6", "gate_passed": true }
}
FieldMeaning
decisionThe answer. Act on this.
valueThe same answer, or null when it's defer. Handy when you store answers.
pProbability of the top class.
probabilitiesEvery class's probability. Useful to see the runner-up.
thresholdThe bar p had to clear for a firm answer.
defer_reasonWhy it deferred, when it did.
decided_by, rulerule when a certain environment rule answered (see below).
modelThe release and model version that answered. Log it.

Confidence and defer

Each task has a threshold (reported in threshold; a class can carry a stricter or looser bar than the task default). When the top class doesn't clear it, the answer is defer rather than a weak guess. defer_reason says why:

defer_reasonMeaningWhat to do
below_thresholdThe evidence points two ways.Send it to a person, or add evidence (history, diff, DOM check) and ask again.
insufficient_evidenceThe request said nothing about this test or locator.Add at least one piece of history, diff or DOM evidence.
model_gate_failedThis task's model didn't pass its release quality gate, so it's answering defer on purpose.Treat as "no answer" for now; it clears with the next release.

A defer answer is billed like any other: it's a real answer ("not sure").

Likely flaky

Run history alone can't always tell a flaky test from an intermittent product bug, so triage has a hedged answer, likely_flaky: the test is probably flaky but the model isn't certain enough to say so. Quarantine and re-run it, but don't close the issue. In the current release, likely_flaky is only given when the request shows the test has passed on a retry at least once (memory.passed_on_retry_50 of 1 or more); otherwise the same evidence defers.

Rule-decided answers

Some failures are environment problems whatever else the context says. When the triage error_message matches one of these, a rule answers environment with p 0.99, decided_by: "rule" and the rule's id:

ruleMatches
browser_not_installed"Executable doesn't exist", "browsers are not installed", "please run the following command to download new browsers"
host_missing_dependencies"Host system is missing dependencies"
webdriver_session_not_created"SessionNotCreated", "session not created"
port_in_useEADDRINUSE
disk_fullENOSPC, "no space left on device"

probabilities still holds the model's own view.

Releases and quality gates

Ostra Decision improves through releases (v.beta.5, for example). Each task's model must pass a quality gate before release; model.gate_passed is true in normal operation. A release can change answers for the same input but never the request or response shape. The live release is in GET /api/v1/models and in the Console under Models → Ostra Decision.

Caching

The same task with the same input, in the same workspace, within 24 hours, is answered from cache: cached: true, no charge. Key order in the JSON doesn't matter; any other difference makes a new request. A new release clears the cache. Cached requests still count toward your rate limit.

Batching

POST /api/v1/batch runs up to 100 tasks in one request, mixed tasks allowed, at half price. Use it whenever you have more than one thing to ask: all the failures of a CI run, all the locators of a page object. Each item answers on its own; one bad item doesn't fail the batch.

Limits

Limit
Request body2 MB
Items per batch100
Candidates per test-impact request2,000
Locators per selector-survival request500
History entries per locator100
error_message8,000 characters
Identifier fields (test_id, run_id, locator_id, …)1–512 characters
Answer timeUsually milliseconds. A request the model hasn't answered in 10 seconds fails with 504 upstream_timeout and isn't charged.

Requests per minute depend on your workspace's usage level.

Where to use it

Each answer has a decision_id. Log it with the test so you can match answers to outcomes later.

Was this page helpful?

On this page