Check selectors before release

Keep a manifest of the locators your tests depend on, check them against a preview build, record their history per release, and fail the pull request when Ostra Decision predicts one will break.

Locators that are about to break show warning signs first: they stop resolving on some runs, or they're tied to layout that keeps changing. This guide checks every locator your tests depend on against a preview build, sends the results and their history to Ostra Decision's selector survival task, and fails the pull request on any breaks.

Cost: 300 locators cost 300 × $0.00000025 = $0.000075 per check.

1. List your locators

Keep a manifest next to your page objects. selector is any Playwright selector (CSS, xpath=…, text=…, role=…, [data-testid=…]) and path is the page it lives on.

tests/selectors.json
[
  { "locator_id": "checkout.apply-coupon", "path": "/checkout", "strategy": "css", "selector": "#apply-coupon" },
  { "locator_id": "checkout.pay-button", "path": "/checkout", "strategy": "test_id", "selector": "[data-testid=pay]" },
  { "locator_id": "nav.account-menu", "path": "/", "strategy": "role", "selector": "role=button[name=\"Account\"]" }
]

strategy is one of css, xpath, test_id, text, role, none, matching how the locator is written in your tests.

2. Add the script

Save as scripts/ostra-selectors.mjs. It needs @playwright/test (already in a Playwright project).

scripts/ostra-selectors.mjs
// Check locators against a build, ask Ostra Decision which will break, keep their history.
// Usage: node scripts/ostra-selectors.mjs check            -> fails on "breaks"
//        node scripts/ostra-selectors.mjs record <release> -> appends this build's results to the history
// Env:   OSTRA_API_KEY, BASE_URL (the build to check, e.g. a preview deployment)
import { chromium } from '@playwright/test';
import { existsSync, readFileSync, writeFileSync } from 'node:fs';

const [mode = 'check', release] = process.argv.slice(2);
const MANIFEST = 'tests/selectors.json';
const HISTORY = 'tests/selector-history.json';
const locators = JSON.parse(readFileSync(MANIFEST, 'utf8'));
const history = existsSync(HISTORY) ? JSON.parse(readFileSync(HISTORY, 'utf8')) : {};

// 1. Which locators resolve in this build?
const browser = await chromium.launch();
const page = await browser.newPage();
const found = {};
for (const path of [...new Set(locators.map((l) => l.path))]) {
  await page.goto(new URL(path, process.env.BASE_URL).toString(), { waitUntil: 'networkidle' });
  for (const l of locators.filter((x) => x.path === path)) {
    found[l.locator_id] = (await page.locator(l.selector).count()) > 0;
  }
}
await browser.close();

if (mode === 'record') {
  if (!release) throw new Error('Usage: record <release>');
  for (const l of locators) {
    history[l.locator_id] = [...(history[l.locator_id] ?? []), { release, resolve_rate: found[l.locator_id] ? 1 : 0 }].slice(-100);
  }
  writeFileSync(HISTORY, `${JSON.stringify(history, null, 2)}\n`);
  console.log(`Recorded ${locators.length} locators for ${release}.`);
  process.exit(0);
}

// 2. Ask Ostra, up to 500 locators per request.
const items = locators.map((l) => ({
  locator_id: l.locator_id,
  strategy: l.strategy,
  dom_locator_found: found[l.locator_id],
  ...(history[l.locator_id]?.length ? { history: history[l.locator_id] } : {}),
}));
const decisions = [];
for (let i = 0; i < items.length; i += 500) {
  const res = await fetch('https://ostra.testrelic.ai/api/v1/predict', {
    method: 'POST',
    headers: { Authorization: `Bearer ${process.env.OSTRA_API_KEY}`, 'Content-Type': 'application/json' },
    body: JSON.stringify({ model: 'ostra-lite', task: 'selector_survival', input: { locators: items.slice(i, i + 500) } }),
  });
  const json = await res.json();
  if (!res.ok) throw new Error(`Ostra ${res.status} ${json.error?.type}: ${json.error?.message}`);
  decisions.push(...json.output.decisions);
}

// 3. Report and fail on "breaks".
const breaking = decisions.filter((d) => d.decision === 'breaks');
for (const d of decisions) console.log(`${d.decision.padEnd(8)} ${String(Math.round(d.p * 100)).padStart(3)}%  ${d.locator_id}`);
if (breaking.length) {
  console.error(`\n${breaking.length} locator(s) likely to break. Prefer role or test-id locators for them.`);
  process.exit(1);
}

3. Run it on pull requests and record it on releases

.github/workflows/selectors.yml
name: selectors
on:
  pull_request:
  release:
    types: [published]

jobs:
  selectors:
    runs-on: ubuntu-latest
    permissions:
      contents: write   # only needed to commit the history on release
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install --with-deps chromium

      - name: Check selectors
        if: github.event_name == 'pull_request'
        run: node scripts/ostra-selectors.mjs check
        env:
          OSTRA_API_KEY: ${{ secrets.OSTRA_API_KEY }}
          BASE_URL: ${{ vars.PREVIEW_URL }}

      - name: Record selector history
        if: github.event_name == 'release'
        run: |
          node scripts/ostra-selectors.mjs record "${{ github.event.release.tag_name }}"
          git config user.name "github-actions"
          git config user.email "github-actions@users.noreply.github.com"
          git add tests/selector-history.json
          git commit -m "Record selector history for ${{ github.event.release.tag_name }}" || true
          git push origin HEAD:${{ github.event.repository.default_branch }}
        env:
          BASE_URL: ${{ vars.PRODUCTION_URL }}

Set PREVIEW_URL (the build a pull request deploys to) and PRODUCTION_URL as repository variables.

The first check works on the DOM result alone. History makes answers much sharper: a locator whose resolve rate falls release after release is the clearest sign of a coming break. If your test runs already know how often each locator resolved, record that rate instead of the 1/0 from a single page load.

What to do with breaks

Replace the locator before release. In order of robustness:

  1. getByRole with an accessible name (role=button[name="Apply coupon"]).
  2. A dedicated test id ([data-testid=apply-coupon]) that the app team agrees not to change.
  3. Visible text, for content that's part of the spec.
  4. CSS or XPath tied to layout: last resort.

defer means there wasn't enough to go on (no history and the locator couldn't be checked) or the evidence was mixed: review those by hand.

Was this page helpful?

On this page