Augur — text-to-image evals

testrelic augur runs evals for a text-to-image pipeline: it drives each eval case's prompt through your generator, judges the output against an approved reference, and renders a Verdict with evidence — SSIM, changed-pixel ratio, and perceptual hash, backed by an HTML report. Augur judges images — it doesn't generate them by default; generation is your own pipeline (or one of the built-in backends, for seeding references).

A hand-recreated transcript of the real pipeline output — switch between demo, run, a multi-model comparison, selfbench, and uploading a verdict.

Try it with no setup

testrelic augur demo --open

Synthesizes 5 scenes and 9 real-world GenAI failure patterns (color drift, hallucinated content, watermark injection, resolution change, sampler noise, compression, framing shift, …), judges them through the full pipeline, and grades detection against calibrated expectations. Exit 0 means every pattern landed on its expected verdict — a good way to confirm the pipeline works before pointing it at your own images.

The real workflow

testrelic augur init                                # scaffold .testrelic-augur/
testrelic augur baseline add ./ref.png --id hero-shot # approve a reference image
testrelic augur run                                  # judge suite.json against references
testrelic augur open                                 # review the evidence report

• augur init scaffolds .testrelic-augur/ (config.json + baselines/) in the current repo.
• augur baseline add|list|show|rm manages the approved-reference store — the golden images every eval case is judged against.
• augur run executes .testrelic-augur/suite.json, generating each selected case's candidate and judging it, producing one verdict for the run.
• augur match <candidate> [golden] judges a single candidate (or directory) directly against the reference store, without a suite — useful for one-off checks.
• augur open [run-dir] opens the latest (or given) run's HTML report connected: an ephemeral 127.0.0.1 review server lets the report's Apply buttons approve references directly. Idles out after 30 minutes; --no-serve opens it as a plain file instead.
• augur report <run-dir> re-renders the HTML report from a finished run directory.

Same commands work from the interactive UI as /augur <action> — /augur run, /augur baseline add <img> --id <test-id>, /augur open, and so on.

Multi-model comparison

A suite.json can carry more than one generator — generators: [{id, backend|command, model}] — unifying built-in SDK backends and your own shell hook. Legacy single-generator suites (generator: {command}) keep working byte-for-byte; a multi-model suite renders every eval case once per generator, so you can compare, say, Gemini against your own pipeline on the identical prompt set.

testrelic augur run --concurrency 8 --attribution

• --concurrency N bounds how many (eval case × generator) generations run at once (default 4).
• Each candidate's baseline reference is a composite id, {testId}__{generatorId}, so the same golden store holds references per model without collision.
• augur baseline --model <generatorId> promotes a generator's candidates to approved references. On a brand-new multi-model suite with no references yet, it falls back to the run's candidates/ directory (matched by the __{generatorId} suffix) instead of requiring a prior verdict — so a first run can seed its own references without hand-adding images one at a time.
• The /augur TUI form is the same: /augur baseline <generatorId>.
• A backend generator with no configured API key skips with a per-column notice, not a run failure.

Generator backends

augur configure --backend gemini|flux|openai stores a backend's API key locally (~/.testrelic/<backend>-key, owner-only; Local mode only — cloud-side key storage isn't supported yet). Env vars always win over a stored key.

testrelic augur configure --backend gemini   # prompts for a masked key
testrelic augur gen "a red running shoe on white" --backend gemini --as-baseline hero-shot

augur gen generates a single image via a configured backend to seed a reference (Augur judges images — generation here is scaffolding, not the eval itself). Backends: gemini (default, also backs imagen), imagen, flux (BFL_API_KEY), openai (OPENAI_API_KEY). --edit <path> turns the prompt into an edit instruction against an existing image instead of generating from scratch.

Semantic attribution (grounding)

--attribution (or suite.json "grounding": {"enabled": true}) grounds bracketed [phrases] in your prompt to regions of the candidate image via Gemini — for example a prompt like a shoe with [a red laptop] on the sole extracts a red laptop as a phrase and locates it in the image. Each phrase is scored honored (confidently present at a location), uncertain, or missing. This is evidence, not verdict — attribution always runs after the verdict has already been decided and can never change it; a per-image grounding failure just leaves that case without an attribution block. Results are cached by image + phrase set, so re-runs are free.

Benchmark the judge itself

testrelic augur selfbench --target 0.99

Runs the verdict engine against a ≥1,000-case labeled synthetic corpus (scene variants × drift ops × graded magnitudes) — "eval the evaluator." Reports overall accuracy with a 95% CI, per-op scores, and a confusion matrix; exits 0 iff accuracy meets --target. Pass --suite to score against your own labeled pairs (suite.json cases with "expected": "pass"|"fail") instead of the synthetic corpus.

Uploading verdicts

testrelic augur run --upload            # judge and upload in one step
testrelic augur upload <run-dir>        # upload a finished run's verdict.json

Uploads to TestRelic evals (Cloud mode + a tr_live_* ingest key). --upload-dry-run prints the exact payload without any HTTP call — useful for checking what a run would send before it sends it. Multi-model runs upload the generator roster, per-case prompt-phrase groups, and any attribution bundle; legacy single-generator suites upload none of that extra shape.

augur run flags

Next steps

→ Modes & engines · Upload existing reports · Artifacts & headless output · Augur in Studio — the same evidence report, plus in-pane review, in a live workbench surface