Decision

Failure triage

Classify a failing test as flaky, likely flaky, product bug, test bug, environment or selector drift — what to send from Playwright, Cypress, Jest or Selenium, what each answer means and what to do next.

Task: triage · MCP tool: ostra_triage_failure · Price: $0.50 per 1M failures ($0.25 in a batch)

Failure triage answers why did this test fail? Call it for every failed test after a CI run, before anyone opens a log: real regressions go to the team, flakes go to quarantine, and broken tests go to whoever owns them.

What to send

input has an optional test_id and run_id (echoed back) and a required context. Inside context, send error_message or error_class, plus as much evidence as you have.

FieldWhat it isWhere to get it
error_messageThe raw error text, up to 8,000 characters. Ostra derives the error class from it.Playwright: result.error.message in the JSON reporter. Cypress: err.message. Jest: the failure message. Selenium: the exception message. Strip ANSI colour codes.
error_classThe normalised class, if you already know it: timeout, locator_not_found, stale_element, strict_mode_violation, assertion_mismatch, network_error, script_error, unmapped.Your own error mapping. Send this or error_message.
frameworkplaywright, cypress, jest or selenium.Your runner.
capture_levelHow much was captured: full (trace, DOM, network), partial, none.Playwright with trace: 'on' → full; screenshots only → partial.
actionThe step that failed: click, fill, navigate, assert, wait_for, api_call, screenshot.The failing step in the trace or stack.
locator.strategyHow the failing step found its element: css, xpath, test_id, text, role, none.The locator in the failing step: getByTestId → test_id, getByRole → role.
diffThe change under test: files_changed, lines_added, lines_removed, touches_test_file, touches_selector_file, locator_values_removed.git diff --numstat <base>...HEAD. locator_values_removed: the diff removes or renames an id, test id, class or text the test uses.
dom.locator_foundWhether the locator still matched an element in the DOM captured at failure time.A Playwright trace or DOM snapshot.
memoryThe test's last 50 runs at most: runs_50, failed_count_50, passed_on_retry_50, consecutive_failures_50, flake_rate_50 (0–1), signature_repeat_count (how often this exact error was seen before).Your test history: the TestRelic platform, your CI's test analytics, or a history file you keep.

At least one of memory, diff or dom must carry something, or the answer is defer (insufficient_evidence). History (memory) is the strongest signal; send it whenever you can.

Example

curl https://ostra.testrelic.ai/api/v1/predict \
  -H "Authorization: Bearer $OSTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ostra-lite",
    "task": "triage",
    "input": {
      "test_id": "checkout.spec.ts > applies coupon",
      "run_id": "gha-18241",
      "context": {
        "framework": "playwright",
        "capture_level": "full",
        "error_message": "TimeoutError: locator.click: Timeout 30000ms exceeded.\n  waiting for locator(\"#apply-coupon\")",
        "action": "click",
        "locator": { "strategy": "css" },
        "diff": { "files_changed": 3, "lines_added": 41, "lines_removed": 12, "touches_test_file": false, "touches_selector_file": false, "locator_values_removed": false },
        "dom": { "locator_found": true },
        "memory": { "runs_50": 40, "flake_rate_50": 0.15, "failed_count_50": 6, "signature_repeat_count": 5 }
      }
    }
  }'
{
  "id": "req_01JDXK4M7Q2N8R5T6V9W3Y1Z0A",
  "model": "ostra-lite@v.beta.5",
  "task": "triage",
  "cached": false,
  "billed": { "units": 1, "costNano": 500, "costUsd": "0.0000005" },
  "output": {
    "decision_id": "dec_5b0c9e2a7f3d4e18a6b2c9d0e1f4a7b3",
    "task": "triage",
    "decision": "flaky",
    "value": "flaky",
    "p": 0.83,
    "probabilities": { "flaky": 0.83, "product_bug": 0.06, "test_bug": 0.05, "environment": 0.04, "selector_drift": 0.02 },
    "threshold": 0.6,
    "defer_reason": null,
    "decided_by": "model",
    "rule": null,
    "model": { "release": "v.beta.5", "version": "1.4.0", "content_hash": "sha256:3f9a1c07d2e4b8a6", "gate_passed": true },
    "test_id": "checkout.spec.ts > applies coupon",
    "run_id": "gha-18241",
    "error_class": "timeout",
    "error_class_source": "error_message"
  }
}
  • decision / p: the answer and its confidence.
  • error_class / error_class_source: the class the model used, and whether it came from your error_class (request) or was derived from error_message.
  • billed: one failure, $0.0000005.

Answers and what to do

decisionMeansNext step
flakyNon-deterministic.Retry, then quarantine if it repeats.
likely_flakyProbably non-deterministic, but an intermittent app bug looks the same.Quarantine and re-run; don't close it.
product_bugThe code under test regressed.File a bug. Keep the build red.
test_bugThe test is wrong or stale.Fix the test.
environmentInfrastructure, network or a dependency.Re-run on a healthy runner; fix the CI setup if it repeats.
selector_driftThe locator no longer matches the page.Update the selector.
deferNot confident enough (see defer_reason).Send it to a person, or add evidence and ask again.

For certain environment errors (a browser that was never installed, missing host dependencies, a WebDriver session that couldn't start, a port in use, a full disk) a rule answers environment directly. See rule-decided answers.

Getting better answers

  • Send history. runs_50, failed_count_50 and flake_rate_50 separate flaky tests from real breaks better than anything else. passed_on_retry_50 enables the likely_flaky answer.
  • Send the diff. A failure on a change that touched the selector file, or removed a locator value, points to selector_drift or test_bug.
  • Send the DOM check. dom.locator_found: false with a removed locator value is strong evidence of drift.
  • Use the raw error message. Don't summarise it; up to 8,000 characters are used.
  • Don't send zeros for unknowns. Leave fields out instead.
  • Batch a run's failures with /batch for half price.

Reference

Was this page helpful?

On this page