Failure triage
Classify a failing test as flaky, likely flaky, product bug, test bug, environment or selector drift — what to send from Playwright, Cypress, Jest or Selenium, what each answer means and what to do next.
Task: triage · MCP tool: ostra_triage_failure · Price: $0.50 per 1M failures ($0.25 in a batch)
Failure triage answers why did this test fail? Call it for every failed test after a CI run, before anyone opens a log: real regressions go to the team, flakes go to quarantine, and broken tests go to whoever owns them.
What to send
input has an optional test_id and run_id (echoed back) and a required context. Inside context, send error_message or error_class, plus as much evidence as you have.
| Field | What it is | Where to get it |
|---|---|---|
error_message | The raw error text, up to 8,000 characters. Ostra derives the error class from it. | Playwright: result.error.message in the JSON reporter. Cypress: err.message. Jest: the failure message. Selenium: the exception message. Strip ANSI colour codes. |
error_class | The normalised class, if you already know it: timeout, locator_not_found, stale_element, strict_mode_violation, assertion_mismatch, network_error, script_error, unmapped. | Your own error mapping. Send this or error_message. |
framework | playwright, cypress, jest or selenium. | Your runner. |
capture_level | How much was captured: full (trace, DOM, network), partial, none. | Playwright with trace: 'on' → full; screenshots only → partial. |
action | The step that failed: click, fill, navigate, assert, wait_for, api_call, screenshot. | The failing step in the trace or stack. |
locator.strategy | How the failing step found its element: css, xpath, test_id, text, role, none. | The locator in the failing step: getByTestId → test_id, getByRole → role. |
diff | The change under test: files_changed, lines_added, lines_removed, touches_test_file, touches_selector_file, locator_values_removed. | git diff --numstat <base>...HEAD. locator_values_removed: the diff removes or renames an id, test id, class or text the test uses. |
dom.locator_found | Whether the locator still matched an element in the DOM captured at failure time. | A Playwright trace or DOM snapshot. |
memory | The test's last 50 runs at most: runs_50, failed_count_50, passed_on_retry_50, consecutive_failures_50, flake_rate_50 (0–1), signature_repeat_count (how often this exact error was seen before). | Your test history: the TestRelic platform, your CI's test analytics, or a history file you keep. |
At least one of memory, diff or dom must carry something, or the answer is defer (insufficient_evidence). History (memory) is the strongest signal; send it whenever you can.
Example
curl https://ostra.testrelic.ai/api/v1/predict \
-H "Authorization: Bearer $OSTRA_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "ostra-lite",
"task": "triage",
"input": {
"test_id": "checkout.spec.ts > applies coupon",
"run_id": "gha-18241",
"context": {
"framework": "playwright",
"capture_level": "full",
"error_message": "TimeoutError: locator.click: Timeout 30000ms exceeded.\n waiting for locator(\"#apply-coupon\")",
"action": "click",
"locator": { "strategy": "css" },
"diff": { "files_changed": 3, "lines_added": 41, "lines_removed": 12, "touches_test_file": false, "touches_selector_file": false, "locator_values_removed": false },
"dom": { "locator_found": true },
"memory": { "runs_50": 40, "flake_rate_50": 0.15, "failed_count_50": 6, "signature_repeat_count": 5 }
}
}
}'{
"id": "req_01JDXK4M7Q2N8R5T6V9W3Y1Z0A",
"model": "ostra-lite@v.beta.5",
"task": "triage",
"cached": false,
"billed": { "units": 1, "costNano": 500, "costUsd": "0.0000005" },
"output": {
"decision_id": "dec_5b0c9e2a7f3d4e18a6b2c9d0e1f4a7b3",
"task": "triage",
"decision": "flaky",
"value": "flaky",
"p": 0.83,
"probabilities": { "flaky": 0.83, "product_bug": 0.06, "test_bug": 0.05, "environment": 0.04, "selector_drift": 0.02 },
"threshold": 0.6,
"defer_reason": null,
"decided_by": "model",
"rule": null,
"model": { "release": "v.beta.5", "version": "1.4.0", "content_hash": "sha256:3f9a1c07d2e4b8a6", "gate_passed": true },
"test_id": "checkout.spec.ts > applies coupon",
"run_id": "gha-18241",
"error_class": "timeout",
"error_class_source": "error_message"
}
}decision/p: the answer and its confidence.error_class/error_class_source: the class the model used, and whether it came from yourerror_class(request) or was derived fromerror_message.billed: one failure, $0.0000005.
Answers and what to do
decision | Means | Next step |
|---|---|---|
flaky | Non-deterministic. | Retry, then quarantine if it repeats. |
likely_flaky | Probably non-deterministic, but an intermittent app bug looks the same. | Quarantine and re-run; don't close it. |
product_bug | The code under test regressed. | File a bug. Keep the build red. |
test_bug | The test is wrong or stale. | Fix the test. |
environment | Infrastructure, network or a dependency. | Re-run on a healthy runner; fix the CI setup if it repeats. |
selector_drift | The locator no longer matches the page. | Update the selector. |
defer | Not confident enough (see defer_reason). | Send it to a person, or add evidence and ask again. |
For certain environment errors (a browser that was never installed, missing host dependencies, a WebDriver session that couldn't start, a port in use, a full disk) a rule answers environment directly. See rule-decided answers.
Getting better answers
- Send history.
runs_50,failed_count_50andflake_rate_50separate flaky tests from real breaks better than anything else.passed_on_retry_50enables thelikely_flakyanswer. - Send the diff. A failure on a change that touched the selector file, or removed a locator value, points to
selector_driftortest_bug. - Send the DOM check.
dom.locator_found: falsewith a removed locator value is strong evidence of drift. - Use the raw error message. Don't summarise it; up to 8,000 characters are used.
- Don't send zeros for unknowns. Leave fields out instead.
- Batch a run's failures with
/batchfor half price.
Reference
- REST:
POST /api/v1/predictwithtask: "triage";POST /api/v1/batch. - MCP:
ostra_triage_failuretakes the sameinputas its arguments. - Guide: Triage failures in CI.
Overview
One fast model with three CI tasks — failure triage, test impact and selector survival. How it decides, when it defers, what it needs from you, and its limits.
Test impact
Give Ostra a code change and the tests that could be affected; get back the tests to run first, ranked. What to send, how selection and fallback work, and how to use the answer in CI.