Triage failures in CI

After every Playwright run in GitHub Actions, send the failures to Ostra Decision in one batch, write a triage table to the job summary, and fail the build only on product bugs.

Playwright run (JSON report)
failed tests
ostra-triage.mjs
up to 100 per call
Ostra Decision /batch
Job summary table
Exit code

This guide adds a triage step to a Playwright job. When tests fail, a small Node script reads the JSON report, adds the code change as evidence, sends every failure to POST /api/v1/batch (half price), writes a table to the GitHub job summary, and fails the job only if a failure is a product bug. Flaky tests and broken tests stop blocking merges; real regressions still do.

Cost: a run with 40 failures costs 40 × $0.00000025 = $0.00001.

Prerequisites

  • An Ostra API key with Full access, stored as the repository secret OSTRA_API_KEY (Settings → Secrets and variables → Actions). See API keys.
  • Credit on the workspace.
  • Node.js 18 or later (for built-in fetch).

1. Add the script

Save this as scripts/ostra-triage.mjs. It has no dependencies.

scripts/ostra-triage.mjs
// Triage the failed tests of a Playwright JSON report with Ostra Decision.
// Usage: node scripts/ostra-triage.mjs <report.json>
// Env:   OSTRA_API_KEY (required), BASE_REF (diff base, default HEAD~1),
//        FAIL_ON (comma-separated decisions that fail the job, default "product_bug")
import { execSync } from 'node:child_process';
import { appendFileSync, readFileSync } from 'node:fs';

const API = 'https://ostra.testrelic.ai/api/v1/batch';
const KEY = process.env.OSTRA_API_KEY;
const FAIL_ON = new Set((process.env.FAIL_ON ?? 'product_bug').split(',').map((s) => s.trim()));
const TEST_FILE = /\.(spec|test)\.[cm]?[jt]sx?$/;
const SELECTOR_FILE = /(^|\/)(pages?|page-objects?|pom|selectors?|locators?)\//i;

if (!KEY) throw new Error('Set OSTRA_API_KEY');
const report = JSON.parse(readFileSync(process.argv[2] ?? 'test-results/results.json', 'utf8'));

// 1. Failed tests from the report (status "unexpected" = failed after all retries).
const stripAnsi = (s) => s.replace(/\u001b\[[0-9;]*m/g, '');
const failures = [];
function walk(suite, titles) {
  for (const spec of suite.specs ?? []) {
    for (const test of spec.tests ?? []) {
      if (test.status !== 'unexpected') continue;
      const last = test.results.at(-1) ?? {};
      const message = stripAnsi(last.error?.message ?? last.errors?.[0]?.message ?? 'Unknown error');
      failures.push({
        id: [spec.file, ...titles, spec.title].filter(Boolean).join(' > '),
        project: test.projectName,
        message: message.slice(0, 8000),
        attempts: test.results.length,
      });
    }
  }
  for (const child of suite.suites ?? []) walk(child, child.title && child.title !== child.file ? [...titles, child.title] : titles);
}
for (const suite of report.suites ?? []) walk(suite, []);
if (failures.length === 0) {
  console.log('No failed tests.');
  process.exit(0);
}

// 2. The code change under test, as evidence.
function diffStats(base) {
  try {
    const rows = execSync(`git diff --numstat ${base}...HEAD`, { encoding: 'utf8' }).trim().split('\n').filter(Boolean);
    const files = rows.map((r) => r.split('\t'));
    return {
      files_changed: files.length,
      lines_added: files.reduce((n, [a]) => n + (Number(a) || 0), 0),
      lines_removed: files.reduce((n, [, r]) => n + (Number(r) || 0), 0),
      touches_test_file: files.some(([, , f]) => TEST_FILE.test(f)),
      touches_selector_file: files.some(([, , f]) => SELECTOR_FILE.test(f)),
    };
  } catch {
    return undefined; // no git history available: send no diff rather than zeros
  }
}
const diff = diffStats(process.env.BASE_REF || 'HEAD~1');

// 3. One triage item per failure, sent 100 at a time.
const items = failures.map((f) => ({
  custom_id: f.id.slice(0, 128),
  task: 'triage',
  input: {
    test_id: f.id.slice(0, 512),
    run_id: process.env.GITHUB_RUN_ID,
    context: {
      framework: 'playwright',
      error_message: f.message,
      ...(diff ? { diff } : {}),
      // Retries in this run that all failed are evidence too.
      ...(f.attempts > 1 ? { memory: { runs_50: f.attempts, failed_count_50: f.attempts, consecutive_failures_50: f.attempts } } : {}),
    },
  },
}));

async function post(body, attempt = 0) {
  const res = await fetch(API, {
    method: 'POST',
    headers: { Authorization: `Bearer ${KEY}`, 'Content-Type': 'application/json' },
    body: JSON.stringify(body),
  });
  if ((res.status === 429 || res.status >= 500) && attempt < 3) {
    const wait = Number(res.headers.get('retry-after')) || 2 ** attempt;
    await new Promise((r) => setTimeout(r, wait * 1000));
    return post(body, attempt + 1);
  }
  const json = await res.json();
  if (!res.ok) throw new Error(`Ostra ${res.status} ${json.error?.type}: ${json.error?.message}`);
  return json;
}

const results = [];
for (let i = 0; i < items.length; i += 100) {
  const batch = await post({ model: 'ostra-lite', items: items.slice(i, i + 100) });
  results.push(...batch.results);
}

// 4. Report.
const NEXT = {
  flaky: 'Retry, then quarantine if it repeats',
  likely_flaky: 'Quarantine and re-run; keep the issue open',
  product_bug: 'File a bug',
  test_bug: 'Fix the test',
  environment: 'Re-run on a healthy runner',
  selector_drift: 'Update the selector',
  defer: 'Needs a person',
};
const rows = results.map((r) => {
  const decision = r.ok ? r.output.decision : 'error';
  const p = r.ok ? `${Math.round(r.output.p * 100)}%` : '';
  const next = r.ok ? NEXT[decision] ?? '' : `${r.error.type}: ${r.error.message}`;
  return { test: r.custom_id, decision, p, next };
});
const table = [
  '### Ostra failure triage',
  '',
  '| Test | Decision | Confidence | Next step |',
  '|---|---|---|---|',
  ...rows.map((r) => `| ${r.test.replace(/\|/g, '\\|')} | \`${r.decision}\` | ${r.p} | ${r.next} |`),
  '',
].join('\n');
console.log(table);
if (process.env.GITHUB_STEP_SUMMARY) appendFileSync(process.env.GITHUB_STEP_SUMMARY, table);

const blocking = rows.filter((r) => FAIL_ON.has(r.decision));
if (blocking.length) {
  console.error(`${blocking.length} failure(s) triaged as ${[...FAIL_ON].join(', ')}.`);
  process.exit(1);
}

2. Add the workflow step

The test step is allowed to fail (continue-on-error), so the triage step decides whether the job fails.

.github/workflows/e2e.yml
name: e2e
on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # full history, for the diff evidence

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - run: npm ci
      - run: npx playwright install --with-deps

      - name: Run tests
        id: tests
        continue-on-error: true
        run: npx playwright test --reporter=line,json
        env:
          PLAYWRIGHT_JSON_OUTPUT_NAME: test-results/results.json

      - name: Triage failures with Ostra
        if: steps.tests.outcome == 'failure'
        run: node scripts/ostra-triage.mjs test-results/results.json
        env:
          OSTRA_API_KEY: ${{ secrets.OSTRA_API_KEY }}
          BASE_REF: origin/${{ github.base_ref }}
          FAIL_ON: product_bug

To also block on broken tests, set FAIL_ON: product_bug,test_bug,selector_drift. To keep the old behaviour (any failure fails the job) but still get the table, add defer,flaky,likely_flaky,environment too.

3. Read the result

The job summary shows one row per failed test:

TestDecisionConfidenceNext step
checkout.spec.ts > applies couponflaky83%Retry, then quarantine if it repeats
orders.api.spec.ts > lists ordersenvironment74%Re-run on a healthy runner
cart.spec.ts > applies taxproduct_bug88%File a bug

Every request also appears in the Console under Logs (channel REST API), with its cost.

Make the answers sharper

The script sends what a single run knows: the error and the code change. Ostra answers more firmly with the test's history. If you keep per-test results (the TestRelic platform does, or a JSON file you cache between runs), add them to context.memory:

memory: {
  runs_50: 40,              // runs in the last 50
  failed_count_50: 6,       // failures among them
  passed_on_retry_50: 3,    // failed, then passed on retry
  flake_rate_50: 0.15,      // share of flaky runs, 0–1
  signature_repeat_count: 5 // times this exact error was seen before
}

Send only what you actually know; leave the rest out. See Failure triage for every field.

JUnit XML instead of Playwright JSON?

Any runner that writes JUnit XML (Cypress, Jest with jest-junit, pytest, Selenium suites) works the same way: each <testcase> with a <failure> child is a failure; use classname and name as the test id and the <failure> message and text as error_message. Set framework to cypress, jest or selenium to match.

Was this page helpful?

On this page