Triage failures in CI
After every Playwright run in GitHub Actions, send the failures to Ostra Decision in one batch, write a triage table to the job summary, and fail the build only on product bugs.
This guide adds a triage step to a Playwright job. When tests fail, a small Node script reads the JSON report, adds the code change as evidence, sends every failure to POST /api/v1/batch (half price), writes a table to the GitHub job summary, and fails the job only if a failure is a product bug. Flaky tests and broken tests stop blocking merges; real regressions still do.
Cost: a run with 40 failures costs 40 × $0.00000025 = $0.00001.
Prerequisites
- An Ostra API key with Full access, stored as the repository secret
OSTRA_API_KEY(Settings → Secrets and variables → Actions). See API keys. - Credit on the workspace.
- Node.js 18 or later (for built-in
fetch).
1. Add the script
Save this as scripts/ostra-triage.mjs. It has no dependencies.
// Triage the failed tests of a Playwright JSON report with Ostra Decision.
// Usage: node scripts/ostra-triage.mjs <report.json>
// Env: OSTRA_API_KEY (required), BASE_REF (diff base, default HEAD~1),
// FAIL_ON (comma-separated decisions that fail the job, default "product_bug")
import { execSync } from 'node:child_process';
import { appendFileSync, readFileSync } from 'node:fs';
const API = 'https://ostra.testrelic.ai/api/v1/batch';
const KEY = process.env.OSTRA_API_KEY;
const FAIL_ON = new Set((process.env.FAIL_ON ?? 'product_bug').split(',').map((s) => s.trim()));
const TEST_FILE = /\.(spec|test)\.[cm]?[jt]sx?$/;
const SELECTOR_FILE = /(^|\/)(pages?|page-objects?|pom|selectors?|locators?)\//i;
if (!KEY) throw new Error('Set OSTRA_API_KEY');
const report = JSON.parse(readFileSync(process.argv[2] ?? 'test-results/results.json', 'utf8'));
// 1. Failed tests from the report (status "unexpected" = failed after all retries).
const stripAnsi = (s) => s.replace(/\u001b\[[0-9;]*m/g, '');
const failures = [];
function walk(suite, titles) {
for (const spec of suite.specs ?? []) {
for (const test of spec.tests ?? []) {
if (test.status !== 'unexpected') continue;
const last = test.results.at(-1) ?? {};
const message = stripAnsi(last.error?.message ?? last.errors?.[0]?.message ?? 'Unknown error');
failures.push({
id: [spec.file, ...titles, spec.title].filter(Boolean).join(' > '),
project: test.projectName,
message: message.slice(0, 8000),
attempts: test.results.length,
});
}
}
for (const child of suite.suites ?? []) walk(child, child.title && child.title !== child.file ? [...titles, child.title] : titles);
}
for (const suite of report.suites ?? []) walk(suite, []);
if (failures.length === 0) {
console.log('No failed tests.');
process.exit(0);
}
// 2. The code change under test, as evidence.
function diffStats(base) {
try {
const rows = execSync(`git diff --numstat ${base}...HEAD`, { encoding: 'utf8' }).trim().split('\n').filter(Boolean);
const files = rows.map((r) => r.split('\t'));
return {
files_changed: files.length,
lines_added: files.reduce((n, [a]) => n + (Number(a) || 0), 0),
lines_removed: files.reduce((n, [, r]) => n + (Number(r) || 0), 0),
touches_test_file: files.some(([, , f]) => TEST_FILE.test(f)),
touches_selector_file: files.some(([, , f]) => SELECTOR_FILE.test(f)),
};
} catch {
return undefined; // no git history available: send no diff rather than zeros
}
}
const diff = diffStats(process.env.BASE_REF || 'HEAD~1');
// 3. One triage item per failure, sent 100 at a time.
const items = failures.map((f) => ({
custom_id: f.id.slice(0, 128),
task: 'triage',
input: {
test_id: f.id.slice(0, 512),
run_id: process.env.GITHUB_RUN_ID,
context: {
framework: 'playwright',
error_message: f.message,
...(diff ? { diff } : {}),
// Retries in this run that all failed are evidence too.
...(f.attempts > 1 ? { memory: { runs_50: f.attempts, failed_count_50: f.attempts, consecutive_failures_50: f.attempts } } : {}),
},
},
}));
async function post(body, attempt = 0) {
const res = await fetch(API, {
method: 'POST',
headers: { Authorization: `Bearer ${KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify(body),
});
if ((res.status === 429 || res.status >= 500) && attempt < 3) {
const wait = Number(res.headers.get('retry-after')) || 2 ** attempt;
await new Promise((r) => setTimeout(r, wait * 1000));
return post(body, attempt + 1);
}
const json = await res.json();
if (!res.ok) throw new Error(`Ostra ${res.status} ${json.error?.type}: ${json.error?.message}`);
return json;
}
const results = [];
for (let i = 0; i < items.length; i += 100) {
const batch = await post({ model: 'ostra-lite', items: items.slice(i, i + 100) });
results.push(...batch.results);
}
// 4. Report.
const NEXT = {
flaky: 'Retry, then quarantine if it repeats',
likely_flaky: 'Quarantine and re-run; keep the issue open',
product_bug: 'File a bug',
test_bug: 'Fix the test',
environment: 'Re-run on a healthy runner',
selector_drift: 'Update the selector',
defer: 'Needs a person',
};
const rows = results.map((r) => {
const decision = r.ok ? r.output.decision : 'error';
const p = r.ok ? `${Math.round(r.output.p * 100)}%` : '';
const next = r.ok ? NEXT[decision] ?? '' : `${r.error.type}: ${r.error.message}`;
return { test: r.custom_id, decision, p, next };
});
const table = [
'### Ostra failure triage',
'',
'| Test | Decision | Confidence | Next step |',
'|---|---|---|---|',
...rows.map((r) => `| ${r.test.replace(/\|/g, '\\|')} | \`${r.decision}\` | ${r.p} | ${r.next} |`),
'',
].join('\n');
console.log(table);
if (process.env.GITHUB_STEP_SUMMARY) appendFileSync(process.env.GITHUB_STEP_SUMMARY, table);
const blocking = rows.filter((r) => FAIL_ON.has(r.decision));
if (blocking.length) {
console.error(`${blocking.length} failure(s) triaged as ${[...FAIL_ON].join(', ')}.`);
process.exit(1);
}2. Add the workflow step
The test step is allowed to fail (continue-on-error), so the triage step decides whether the job fails.
name: e2e
on: [pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # full history, for the diff evidence
- uses: actions/setup-node@v4
with:
node-version: 20
- run: npm ci
- run: npx playwright install --with-deps
- name: Run tests
id: tests
continue-on-error: true
run: npx playwright test --reporter=line,json
env:
PLAYWRIGHT_JSON_OUTPUT_NAME: test-results/results.json
- name: Triage failures with Ostra
if: steps.tests.outcome == 'failure'
run: node scripts/ostra-triage.mjs test-results/results.json
env:
OSTRA_API_KEY: ${{ secrets.OSTRA_API_KEY }}
BASE_REF: origin/${{ github.base_ref }}
FAIL_ON: product_bugTo also block on broken tests, set FAIL_ON: product_bug,test_bug,selector_drift. To keep the old behaviour (any failure fails the job) but still get the table, add defer,flaky,likely_flaky,environment too.
3. Read the result
The job summary shows one row per failed test:
| Test | Decision | Confidence | Next step |
|---|---|---|---|
| checkout.spec.ts > applies coupon | flaky | 83% | Retry, then quarantine if it repeats |
| orders.api.spec.ts > lists orders | environment | 74% | Re-run on a healthy runner |
| cart.spec.ts > applies tax | product_bug | 88% | File a bug |
Every request also appears in the Console under Logs (channel REST API), with its cost.
Make the answers sharper
The script sends what a single run knows: the error and the code change. Ostra answers more firmly with the test's history. If you keep per-test results (the TestRelic platform does, or a JSON file you cache between runs), add them to context.memory:
memory: {
runs_50: 40, // runs in the last 50
failed_count_50: 6, // failures among them
passed_on_retry_50: 3, // failed, then passed on retry
flake_rate_50: 0.15, // share of flaky runs, 0–1
signature_repeat_count: 5 // times this exact error was seen before
}Send only what you actually know; leave the rest out. See Failure triage for every field.
JUnit XML instead of Playwright JSON?
Any runner that writes JUnit XML (Cypress, Jest with jest-junit, pytest, Selenium suites) works the same way: each <testcase> with a <failure> child is a failure; use classname and name as the test id and the <failure> message and text as error_message. Set framework to cypress, jest or selenium to match.
Related
- Failure triage: fields and answers.
POST /api/v1/batch: the endpoint used here.- Run only impacted tests: the next step for faster pipelines.
Pro
TestRelic's testing model for writing and fixing tests, analysing failures and logs, reading screenshots and video, and planning coverage — limits, thinking, tools, images and video, streaming, sessions, cold starts, compatibility and pricing.
Impacted tests
Compute a change's diff and each test's co-change rate from git, ask Ostra Decision which tests the change affects, and run those first with Playwright — falling back to the full suite when there's no evidence.