Run only impacted tests
Compute a change's diff and each test's co-change rate from git, ask Ostra Decision which tests the change affects, and run those first with Playwright — falling back to the full suite when there's no evidence.
On every push, run the tests a change is most likely to break first, and get a red or green signal in a fraction of the time. Each test-impact request costs $0.000002, whatever the number of candidates (up to 2,000).
This guide treats each test file as a candidate, because Playwright can run a list of files directly. You can use test titles with --grep instead.
1. Add the script
Save as scripts/ostra-impact.mjs. It prints the files to run, one per line, or nothing when the full suite should run.
// Ask Ostra Decision which test files a change affects.
// Usage: node scripts/ostra-impact.mjs > impacted.txt
// Env: OSTRA_API_KEY (required), BASE_REF (default HEAD~1), TEST_DIR (default "tests")
import { execSync } from 'node:child_process';
const KEY = process.env.OSTRA_API_KEY;
const BASE = process.env.BASE_REF || 'HEAD~1';
const TEST_DIR = process.env.TEST_DIR || 'tests';
const TEST_FILE = /\.(spec|test)\.[cm]?[jt]sx?$/;
const SELECTOR_FILE = /(^|\/)(pages?|page-objects?|pom|selectors?|locators?)\//i;
const git = (args) => execSync(`git ${args}`, { encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 });
if (!KEY) throw new Error('Set OSTRA_API_KEY');
// 1. The change.
const changed = git(`diff --numstat ${BASE}...HEAD`).trim().split('\n').filter(Boolean).map((r) => r.split('\t'));
const changedFiles = changed.map(([, , f]) => f);
const diff = {
files_changed: changed.length,
lines_added: changed.reduce((n, [a]) => n + (Number(a) || 0), 0),
lines_removed: changed.reduce((n, [, r]) => n + (Number(r) || 0), 0),
touches_test_file: changedFiles.some((f) => TEST_FILE.test(f)),
touches_selector_file: changedFiles.some((f) => SELECTOR_FILE.test(f)),
};
if (changed.length === 0) process.exit(0); // nothing changed: run nothing extra
// 2. Candidates: every test file, with how often it changed together with the changed files.
const tests = git(`ls-files ${TEST_DIR}`).trim().split('\n').filter((f) => TEST_FILE.test(f));
const commits = git('log -n 500 --format=--%H --name-only')
.split('--')
.map((block) => new Set(block.trim().split('\n').slice(1).filter(Boolean)));
const touchingChange = commits.filter((files) => changedFiles.some((f) => files.has(f)));
const candidates = tests.slice(0, 2000).map((test_id) => {
const together = touchingChange.filter((files) => files.has(test_id)).length;
const rate = touchingChange.length ? together / touchingChange.length : 0;
return rate > 0 ? { test_id, co_change_rate: Number(rate.toFixed(4)) } : { test_id };
});
// 3. Ask Ostra.
const res = await fetch('https://ostra.testrelic.ai/api/v1/predict', {
method: 'POST',
headers: { Authorization: `Bearer ${KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify({ model: 'ostra-lite', task: 'test_impact', input: { diff, candidates } }),
});
const json = await res.json();
if (!res.ok) {
// Don't block the pipeline on Ostra: an empty list means "run the full suite".
console.error(`Ostra ${res.status} ${json.error?.type}: ${json.error?.message}`);
process.exit(0);
}
const { impacted, selection } = json.output;
console.error(`Ostra selection: ${selection}, ${impacted.length} of ${candidates.length} test files`);
if (selection !== 'none') console.log(impacted.join('\n'));Notes:
- Candidates with no co-change get no rate at all, rather than
0: missing evidence is left out, not zeroed. - If you track per-test failure rates, add
historical_fail_rateto each candidate. - On any error the script prints nothing, so the job falls back to the full suite.
2. Use it in the workflow
name: e2e
on: [pull_request]
jobs:
impacted-first:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # history for the diff and co-change rates
- uses: actions/setup-node@v4
with:
node-version: 20
- run: npm ci
- run: npx playwright install --with-deps
- name: Pick impacted tests
run: node scripts/ostra-impact.mjs > impacted.txt
env:
OSTRA_API_KEY: ${{ secrets.OSTRA_API_KEY }}
BASE_REF: origin/${{ github.base_ref }}
- name: Run impacted tests (or everything)
run: |
if [ -s impacted.txt ]; then
echo "Running $(wc -l < impacted.txt) impacted test file(s)"
xargs npx playwright test < impacted.txt
else
echo "No selection: running the full suite"
npx playwright test
fiselection decides what runs:
selection | What runs |
|---|---|
model | The files Ostra answered impacted, best first. |
fallback_top_candidates | Up to 5 best-ranked files that have co-change evidence. |
none | The full suite (the script prints nothing). |
3. Keep the full suite honest
Selective runs catch most breaks early, not all. Keep running the full suite on a schedule or before release:
on:
pull_request:
schedule:
- cron: '0 3 * * *' # nightly full runand in the run step, skip the selection when github.event_name == 'schedule'.
Using test titles instead of files
To select individual tests, use titles as test_id (list them with npx playwright test --list --reporter=json), and run the answer with --grep, escaping regular-expression characters in each title:
const escape = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
console.log(impacted.map(escape).join('|')); // then: npx playwright test --grep "$(cat impacted.txt)"Related
- Test impact: fields, selection and fallback.
POST /api/v1/predict.- Triage failures in CI.
CI triage
After every Playwright run in GitHub Actions, send the failures to Ostra Decision in one batch, write a triage table to the job summary, and fail the build only on product bugs.
Selector checks
Keep a manifest of the locators your tests depend on, check them against a preview build, record their history per release, and fail the pull request when Ostra Decision predicts one will break.