Run only impacted tests

Compute a change's diff and each test's co-change rate from git, ask Ostra Decision which tests the change affects, and run those first with Playwright — falling back to the full suite when there's no evidence.

git diff
git log
diff stats, co-change
ostra-impact.mjs
one request
Ostra Decision test_impact
impacted, selection
Playwright: impacted files
Full suite (fallback)

On every push, run the tests a change is most likely to break first, and get a red or green signal in a fraction of the time. Each test-impact request costs $0.000002, whatever the number of candidates (up to 2,000).

This guide treats each test file as a candidate, because Playwright can run a list of files directly. You can use test titles with --grep instead.

1. Add the script

Save as scripts/ostra-impact.mjs. It prints the files to run, one per line, or nothing when the full suite should run.

scripts/ostra-impact.mjs
// Ask Ostra Decision which test files a change affects.
// Usage: node scripts/ostra-impact.mjs > impacted.txt
// Env:   OSTRA_API_KEY (required), BASE_REF (default HEAD~1), TEST_DIR (default "tests")
import { execSync } from 'node:child_process';

const KEY = process.env.OSTRA_API_KEY;
const BASE = process.env.BASE_REF || 'HEAD~1';
const TEST_DIR = process.env.TEST_DIR || 'tests';
const TEST_FILE = /\.(spec|test)\.[cm]?[jt]sx?$/;
const SELECTOR_FILE = /(^|\/)(pages?|page-objects?|pom|selectors?|locators?)\//i;
const git = (args) => execSync(`git ${args}`, { encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 });

if (!KEY) throw new Error('Set OSTRA_API_KEY');

// 1. The change.
const changed = git(`diff --numstat ${BASE}...HEAD`).trim().split('\n').filter(Boolean).map((r) => r.split('\t'));
const changedFiles = changed.map(([, , f]) => f);
const diff = {
  files_changed: changed.length,
  lines_added: changed.reduce((n, [a]) => n + (Number(a) || 0), 0),
  lines_removed: changed.reduce((n, [, r]) => n + (Number(r) || 0), 0),
  touches_test_file: changedFiles.some((f) => TEST_FILE.test(f)),
  touches_selector_file: changedFiles.some((f) => SELECTOR_FILE.test(f)),
};
if (changed.length === 0) process.exit(0); // nothing changed: run nothing extra

// 2. Candidates: every test file, with how often it changed together with the changed files.
const tests = git(`ls-files ${TEST_DIR}`).trim().split('\n').filter((f) => TEST_FILE.test(f));
const commits = git('log -n 500 --format=--%H --name-only')
  .split('--')
  .map((block) => new Set(block.trim().split('\n').slice(1).filter(Boolean)));
const touchingChange = commits.filter((files) => changedFiles.some((f) => files.has(f)));
const candidates = tests.slice(0, 2000).map((test_id) => {
  const together = touchingChange.filter((files) => files.has(test_id)).length;
  const rate = touchingChange.length ? together / touchingChange.length : 0;
  return rate > 0 ? { test_id, co_change_rate: Number(rate.toFixed(4)) } : { test_id };
});

// 3. Ask Ostra.
const res = await fetch('https://ostra.testrelic.ai/api/v1/predict', {
  method: 'POST',
  headers: { Authorization: `Bearer ${KEY}`, 'Content-Type': 'application/json' },
  body: JSON.stringify({ model: 'ostra-lite', task: 'test_impact', input: { diff, candidates } }),
});
const json = await res.json();
if (!res.ok) {
  // Don't block the pipeline on Ostra: an empty list means "run the full suite".
  console.error(`Ostra ${res.status} ${json.error?.type}: ${json.error?.message}`);
  process.exit(0);
}

const { impacted, selection } = json.output;
console.error(`Ostra selection: ${selection}, ${impacted.length} of ${candidates.length} test files`);
if (selection !== 'none') console.log(impacted.join('\n'));

Notes:

  • Candidates with no co-change get no rate at all, rather than 0: missing evidence is left out, not zeroed.
  • If you track per-test failure rates, add historical_fail_rate to each candidate.
  • On any error the script prints nothing, so the job falls back to the full suite.

2. Use it in the workflow

.github/workflows/e2e.yml
name: e2e
on: [pull_request]

jobs:
  impacted-first:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # history for the diff and co-change rates

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - run: npm ci
      - run: npx playwright install --with-deps

      - name: Pick impacted tests
        run: node scripts/ostra-impact.mjs > impacted.txt
        env:
          OSTRA_API_KEY: ${{ secrets.OSTRA_API_KEY }}
          BASE_REF: origin/${{ github.base_ref }}

      - name: Run impacted tests (or everything)
        run: |
          if [ -s impacted.txt ]; then
            echo "Running $(wc -l < impacted.txt) impacted test file(s)"
            xargs npx playwright test < impacted.txt
          else
            echo "No selection: running the full suite"
            npx playwright test
          fi

selection decides what runs:

selectionWhat runs
modelThe files Ostra answered impacted, best first.
fallback_top_candidatesUp to 5 best-ranked files that have co-change evidence.
noneThe full suite (the script prints nothing).

3. Keep the full suite honest

Selective runs catch most breaks early, not all. Keep running the full suite on a schedule or before release:

on:
  pull_request:
  schedule:
    - cron: '0 3 * * *'   # nightly full run

and in the run step, skip the selection when github.event_name == 'schedule'.

Using test titles instead of files

To select individual tests, use titles as test_id (list them with npx playwright test --list --reporter=json), and run the answer with --grep, escaping regular-expression characters in each title:

const escape = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
console.log(impacted.map(escape).join('|'));   // then: npx playwright test --grep "$(cat impacted.txt)"
Was this page helpful?

On this page