Test impact
Give Ostra a code change and the tests that could be affected; get back the tests to run first, ranked. What to send, how selection and fallback work, and how to use the answer in CI.
Task: test_impact · MCP tool: ostra_select_impacted_tests · Price: $2.00 per 1M requests ($1.00 in a batch), whatever the number of candidates
Test impact answers which tests does this change affect? Use it to run the tests most likely to break first, or to run only those on every push and the full suite less often.
What to send
| Field | What it is | Where to get it |
|---|---|---|
diff.files_changed | Files changed. | git diff --numstat <base>...HEAD | wc -l |
diff.lines_added, diff.lines_removed | Lines added and removed. | Sum the two numeric columns of git diff --numstat. |
diff.touches_test_file | The change edits a test file. | Any changed path matching your test pattern (*.spec.ts, *.test.js, *.cy.ts, test_*.py). |
diff.touches_selector_file | The change edits a page object, locator map or other selector file. | Any changed path under your page-object or selector folders. |
candidates[].test_id | A test that could be affected (required, up to 2,000). | Your test files or test ids. Use whatever unit you can run selectively: a file, or a title for --grep. |
candidates[].co_change_rate | How often this test's files change together with the changed files, 0–1. | git log --name-only: of the commits that touched the changed files, the share that also touched this test. |
candidates[].historical_fail_rate | How often this test fails, 0–1. | Your test history. |
A candidate with neither rate (or both 0) carries no evidence. It can still be ranked, but it won't be picked by the fallback described below.
Example
curl https://ostra.testrelic.ai/api/v1/predict \
-H "Authorization: Bearer $OSTRA_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "ostra-lite",
"task": "test_impact",
"input": {
"diff": { "files_changed": 2, "lines_added": 87, "lines_removed": 14, "touches_test_file": false, "touches_selector_file": false },
"candidates": [
{ "test_id": "checkout.spec.ts > applies coupon", "co_change_rate": 0.62, "historical_fail_rate": 0.08 },
{ "test_id": "checkout.spec.ts > rejects expired coupon", "co_change_rate": 0.58, "historical_fail_rate": 0.03 },
{ "test_id": "search.spec.ts > filters by brand", "co_change_rate": 0.02, "historical_fail_rate": 0.04 }
]
}
}'output (trimmed: each decision also carries the common fields shown in Reading an answer):
{
"decisions": [
{ "test_id": "checkout.spec.ts > applies coupon", "decision": "impacted", "p": 0.91, "probabilities": { "impacted": 0.91, "not_impacted": 0.09 } },
{ "test_id": "checkout.spec.ts > rejects expired coupon", "decision": "impacted", "p": 0.86, "probabilities": { "impacted": 0.86, "not_impacted": 0.14 } },
{ "test_id": "search.spec.ts > filters by brand", "decision": "not_impacted", "p": 0.97, "probabilities": { "impacted": 0.03, "not_impacted": 0.97 } }
],
"impacted": ["checkout.spec.ts > applies coupon", "checkout.spec.ts > rejects expired coupon"],
"selection": "model"
}Reading the answer
decisionshas one entry per candidate, sorted by probability ofimpacted, highest first (ties by co-change rate, thentest_id, so the order never depends on the order you sent them). Each isimpacted,not_impactedordefer.impactedis the list to run, best first.selectionsays howimpactedwas chosen:
selection | Means | What to do |
|---|---|---|
model | impacted holds the candidates the model answered impacted. | Run them first (or only them). |
fallback_top_candidates | No candidate cleared the threshold, so up to 5 best-ranked candidates that carry evidence are returned instead. | Run them, and treat the result as a hint rather than a guarantee. |
none | No candidate carries any evidence. impacted is empty. | Run the full suite. |
The model is tuned to catch breaking tests, so a candidate can be answered impacted with a probability below 0.5: compare p with each decision's threshold.
Using it in CI
A common pattern:
- On every push, run
impactedfirst. If any of them fail, report early. - Run the rest of the suite afterwards, or nightly.
- When
selectionisnone, run everything.
Run only impacted tests has a complete GitHub Actions setup that computes the diff and co-change rates from git and runs the selection with Playwright.
Getting better answers
- Send co-change rates. They're the strongest signal, and cheap to compute from
git log. - Send failure rates if your test history has them.
- Pick candidates you can actually run selectively: test files work well; so do test titles with
--grep. - One request per change. You pay per request, not per candidate, so include every candidate (up to 2,000).
Reference
- REST:
POST /api/v1/predictwithtask: "test_impact". - MCP:
ostra_select_impacted_tests.
Failure triage
Classify a failing test as flaky, likely flaky, product bug, test bug, environment or selector drift — what to send from Playwright, Cypress, Jest or Selenium, what each answer means and what to do next.
Selector survival
Predict which UI locators will break in the next release before they break your tests — what to send, what the answers mean, and how to act on them.