Decision

Test impact

Give Ostra a code change and the tests that could be affected; get back the tests to run first, ranked. What to send, how selection and fallback work, and how to use the answer in CI.

Task: test_impact · MCP tool: ostra_select_impacted_tests · Price: $2.00 per 1M requests ($1.00 in a batch), whatever the number of candidates

Test impact answers which tests does this change affect? Use it to run the tests most likely to break first, or to run only those on every push and the full suite less often.

What to send

FieldWhat it isWhere to get it
diff.files_changedFiles changed.git diff --numstat <base>...HEAD | wc -l
diff.lines_added, diff.lines_removedLines added and removed.Sum the two numeric columns of git diff --numstat.
diff.touches_test_fileThe change edits a test file.Any changed path matching your test pattern (*.spec.ts, *.test.js, *.cy.ts, test_*.py).
diff.touches_selector_fileThe change edits a page object, locator map or other selector file.Any changed path under your page-object or selector folders.
candidates[].test_idA test that could be affected (required, up to 2,000).Your test files or test ids. Use whatever unit you can run selectively: a file, or a title for --grep.
candidates[].co_change_rateHow often this test's files change together with the changed files, 0–1.git log --name-only: of the commits that touched the changed files, the share that also touched this test.
candidates[].historical_fail_rateHow often this test fails, 0–1.Your test history.

A candidate with neither rate (or both 0) carries no evidence. It can still be ranked, but it won't be picked by the fallback described below.

Example

curl https://ostra.testrelic.ai/api/v1/predict \
  -H "Authorization: Bearer $OSTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ostra-lite",
    "task": "test_impact",
    "input": {
      "diff": { "files_changed": 2, "lines_added": 87, "lines_removed": 14, "touches_test_file": false, "touches_selector_file": false },
      "candidates": [
        { "test_id": "checkout.spec.ts > applies coupon", "co_change_rate": 0.62, "historical_fail_rate": 0.08 },
        { "test_id": "checkout.spec.ts > rejects expired coupon", "co_change_rate": 0.58, "historical_fail_rate": 0.03 },
        { "test_id": "search.spec.ts > filters by brand", "co_change_rate": 0.02, "historical_fail_rate": 0.04 }
      ]
    }
  }'

output (trimmed: each decision also carries the common fields shown in Reading an answer):

{
  "decisions": [
    { "test_id": "checkout.spec.ts > applies coupon", "decision": "impacted", "p": 0.91, "probabilities": { "impacted": 0.91, "not_impacted": 0.09 } },
    { "test_id": "checkout.spec.ts > rejects expired coupon", "decision": "impacted", "p": 0.86, "probabilities": { "impacted": 0.86, "not_impacted": 0.14 } },
    { "test_id": "search.spec.ts > filters by brand", "decision": "not_impacted", "p": 0.97, "probabilities": { "impacted": 0.03, "not_impacted": 0.97 } }
  ],
  "impacted": ["checkout.spec.ts > applies coupon", "checkout.spec.ts > rejects expired coupon"],
  "selection": "model"
}

Reading the answer

  • decisions has one entry per candidate, sorted by probability of impacted, highest first (ties by co-change rate, then test_id, so the order never depends on the order you sent them). Each is impacted, not_impacted or defer.
  • impacted is the list to run, best first.
  • selection says how impacted was chosen:
selectionMeansWhat to do
modelimpacted holds the candidates the model answered impacted.Run them first (or only them).
fallback_top_candidatesNo candidate cleared the threshold, so up to 5 best-ranked candidates that carry evidence are returned instead.Run them, and treat the result as a hint rather than a guarantee.
noneNo candidate carries any evidence. impacted is empty.Run the full suite.

The model is tuned to catch breaking tests, so a candidate can be answered impacted with a probability below 0.5: compare p with each decision's threshold.

Using it in CI

A common pattern:

  1. On every push, run impacted first. If any of them fail, report early.
  2. Run the rest of the suite afterwards, or nightly.
  3. When selection is none, run everything.

Run only impacted tests has a complete GitHub Actions setup that computes the diff and co-change rates from git and runs the selection with Playwright.

Getting better answers

  • Send co-change rates. They're the strongest signal, and cheap to compute from git log.
  • Send failure rates if your test history has them.
  • Pick candidates you can actually run selectively: test files work well; so do test titles with --grep.
  • One request per change. You pay per request, not per candidate, so include every candidate (up to 2,000).

Reference

Was this page helpful?

On this page