Check selectors before release
Keep a manifest of the locators your tests depend on, check them against a preview build, record their history per release, and fail the pull request when Ostra Decision predicts one will break.
Locators that are about to break show warning signs first: they stop resolving on some runs, or they're tied to layout that keeps changing. This guide checks every locator your tests depend on against a preview build, sends the results and their history to Ostra Decision's selector survival task, and fails the pull request on any breaks.
Cost: 300 locators cost 300 × $0.00000025 = $0.000075 per check.
1. List your locators
Keep a manifest next to your page objects. selector is any Playwright selector (CSS, xpath=…, text=…, role=…, [data-testid=…]) and path is the page it lives on.
[
{ "locator_id": "checkout.apply-coupon", "path": "/checkout", "strategy": "css", "selector": "#apply-coupon" },
{ "locator_id": "checkout.pay-button", "path": "/checkout", "strategy": "test_id", "selector": "[data-testid=pay]" },
{ "locator_id": "nav.account-menu", "path": "/", "strategy": "role", "selector": "role=button[name=\"Account\"]" }
]strategy is one of css, xpath, test_id, text, role, none, matching how the locator is written in your tests.
2. Add the script
Save as scripts/ostra-selectors.mjs. It needs @playwright/test (already in a Playwright project).
// Check locators against a build, ask Ostra Decision which will break, keep their history.
// Usage: node scripts/ostra-selectors.mjs check -> fails on "breaks"
// node scripts/ostra-selectors.mjs record <release> -> appends this build's results to the history
// Env: OSTRA_API_KEY, BASE_URL (the build to check, e.g. a preview deployment)
import { chromium } from '@playwright/test';
import { existsSync, readFileSync, writeFileSync } from 'node:fs';
const [mode = 'check', release] = process.argv.slice(2);
const MANIFEST = 'tests/selectors.json';
const HISTORY = 'tests/selector-history.json';
const locators = JSON.parse(readFileSync(MANIFEST, 'utf8'));
const history = existsSync(HISTORY) ? JSON.parse(readFileSync(HISTORY, 'utf8')) : {};
// 1. Which locators resolve in this build?
const browser = await chromium.launch();
const page = await browser.newPage();
const found = {};
for (const path of [...new Set(locators.map((l) => l.path))]) {
await page.goto(new URL(path, process.env.BASE_URL).toString(), { waitUntil: 'networkidle' });
for (const l of locators.filter((x) => x.path === path)) {
found[l.locator_id] = (await page.locator(l.selector).count()) > 0;
}
}
await browser.close();
if (mode === 'record') {
if (!release) throw new Error('Usage: record <release>');
for (const l of locators) {
history[l.locator_id] = [...(history[l.locator_id] ?? []), { release, resolve_rate: found[l.locator_id] ? 1 : 0 }].slice(-100);
}
writeFileSync(HISTORY, `${JSON.stringify(history, null, 2)}\n`);
console.log(`Recorded ${locators.length} locators for ${release}.`);
process.exit(0);
}
// 2. Ask Ostra, up to 500 locators per request.
const items = locators.map((l) => ({
locator_id: l.locator_id,
strategy: l.strategy,
dom_locator_found: found[l.locator_id],
...(history[l.locator_id]?.length ? { history: history[l.locator_id] } : {}),
}));
const decisions = [];
for (let i = 0; i < items.length; i += 500) {
const res = await fetch('https://ostra.testrelic.ai/api/v1/predict', {
method: 'POST',
headers: { Authorization: `Bearer ${process.env.OSTRA_API_KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify({ model: 'ostra-lite', task: 'selector_survival', input: { locators: items.slice(i, i + 500) } }),
});
const json = await res.json();
if (!res.ok) throw new Error(`Ostra ${res.status} ${json.error?.type}: ${json.error?.message}`);
decisions.push(...json.output.decisions);
}
// 3. Report and fail on "breaks".
const breaking = decisions.filter((d) => d.decision === 'breaks');
for (const d of decisions) console.log(`${d.decision.padEnd(8)} ${String(Math.round(d.p * 100)).padStart(3)}% ${d.locator_id}`);
if (breaking.length) {
console.error(`\n${breaking.length} locator(s) likely to break. Prefer role or test-id locators for them.`);
process.exit(1);
}3. Run it on pull requests and record it on releases
name: selectors
on:
pull_request:
release:
types: [published]
jobs:
selectors:
runs-on: ubuntu-latest
permissions:
contents: write # only needed to commit the history on release
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
- run: npm ci
- run: npx playwright install --with-deps chromium
- name: Check selectors
if: github.event_name == 'pull_request'
run: node scripts/ostra-selectors.mjs check
env:
OSTRA_API_KEY: ${{ secrets.OSTRA_API_KEY }}
BASE_URL: ${{ vars.PREVIEW_URL }}
- name: Record selector history
if: github.event_name == 'release'
run: |
node scripts/ostra-selectors.mjs record "${{ github.event.release.tag_name }}"
git config user.name "github-actions"
git config user.email "github-actions@users.noreply.github.com"
git add tests/selector-history.json
git commit -m "Record selector history for ${{ github.event.release.tag_name }}" || true
git push origin HEAD:${{ github.event.repository.default_branch }}
env:
BASE_URL: ${{ vars.PRODUCTION_URL }}Set PREVIEW_URL (the build a pull request deploys to) and PRODUCTION_URL as repository variables.
The first check works on the DOM result alone. History makes answers much sharper: a locator whose resolve rate falls release after release is the clearest sign of a coming break. If your test runs already know how often each locator resolved, record that rate instead of the 1/0 from a single page load.
What to do with breaks
Replace the locator before release. In order of robustness:
getByRolewith an accessible name (role=button[name="Apply coupon"]).- A dedicated test id (
[data-testid=apply-coupon]) that the app team agrees not to change. - Visible text, for content that's part of the spec.
- CSS or XPath tied to layout: last resort.
defer means there wasn't enough to go on (no history and the locator couldn't be checked) or the evidence was mixed: review those by hand.
Related
- Selector survival: fields and answers.
POST /api/v1/predict.
Impacted tests
Compute a change's diff and each test's co-change rate from git, ask Ostra Decision which tests the change affects, and run those first with Playwright — falling back to the full suite when there's no evidence.
Coding agents
Connect Claude Code, Cursor or VS Code to the Ostra MCP server, then teach your agent when to triage a failure, pick impacted tests, check selectors and ask Ostra Pro — with ready-made CLAUDE.md and Cursor rules.