Augur Multi-Model Comparison

Augur is TestRelic's workspace for text-to-image visual-regression runs — eval runs uploaded by the TestRelic CLI that compare a generated image against expected bounds and render a ship/no-ship verdict. Augur runs are a specialized kind of eval run: they open the same Run Detail page as any other run, but render a dedicated ship-gate workspace instead of the standard case table.

This page covers the multi-model comparison workspace: when a run's cases carry more than one generator model for the same prompts, Augur replaces the single-column case grid with a prompts × models triage matrix, a compare canvas for drilling into one prompt at a time, and prompt-phrase attribution that maps individual words in the prompt to what each model actually painted.

Prompt groups and generator models

Multi-model comparison is driven by two fields the CLI can attach to each eval case it uploads: a prompt group (the same prompt run against several models) and a generator model (which model produced that case's image). Cases that share a prompt group are shown as one row in the matrix, with one column per generator model.

Runs uploaded before this grouping existed have neither field set. Augur treats every case in those runs as its own one-case prompt group, so legacy runs still render — as a one-column matrix with the same data as the classic card grid, just in the triage layout.

The triage matrix

Open a multi-model Augur run and its Run Detail page leads with the ship-gate KPI strip: the run's overall verdict (Ship / Needs Review / No-Ship, taken as the worst case verdict), prompt count, model count, failing-cell count, run duration, and — for multi-model runs — an Agreement tile showing the percentage of prompts where every model landed on the same verdict.

Below the strip, multi-model runs get a model scoreboard: one card per generator model with its pass/review/fail split as a stacked bar. The model with the fewest fails (ties broken by fewest reviews, then most passes) gets a highlighted rail — there's no ranking beyond that, just a quick answer to "which model should we use?"

The prompt × model matrix itself lists one row per prompt group and one column per model:

• Rows with models that disagree on verdict get a SPLIT badge.
• Each cell shows a thumbnail, a colored verdict rail, a colorblind-safe glyph (never color alone), and a review dot — cells with no result render as a dashed placeholder that never reads as a failure.
• Filter chips: All / Fail / Review / Pass, plus Disagreement on multi-model runs.
• Sort: Severity, Disagreement, or Prompt order (Disagreement is hidden for single-model runs, since every row agrees with itself).
• Density: Comfortable or Compact — the matrix defaults to Compact automatically past 40 prompts.
• A Cards | Matrix toggle lets you fall back to the familiar per-case card grid; your choice persists across runs.
• Arrow keys move focus across the grid two-dimensionally, so you can scan a large matrix without a mouse.

Clicking a cell on a single-model run opens the existing case detail view, unchanged. On a multi-model run, it opens the compare canvas at that prompt, focused on that model.

The compare canvas

The compare canvas is the matrix's drill-in level: one prompt, every model, side by side. It opens at ?prompt=<group>&model=<slug> — a shareable, deep-linkable URL — and a legacy ?caseId= link on a multi-model run resolves to the right prompt group and model automatically, so old links and Ask-AI artifacts keep working.

• A backbar shows the prompt as a blockquote, with [ / ] keys (or on-screen controls) to step to the previous/next prompt and Esc to back out to the matrix.
• Grid view lays out one column per model: an accent-colored header with the verdict badge (click to focus that model), the candidate image, a similarity/pixel-diff row, and Approve / Reject review actions.
• Pair view swaps in a wipe-style comparison slider between any two models' candidates — no precomputed diff data required, just the two images.
• Below the columns, the focused model's full evidence renders through the same case-detail view used elsewhere in Augur (why-verdict summary, where-changed regions, technical detail, history) — reused in an embedded mode that hides only the prompt blockquote, since the canvas already shows it.

Prompt-phrase attribution

Attribution is what makes the matrix answer why a model's output failed, not just whether it did. When a run's cases carry phrase-level grounding data, the compare canvas's prompt inspector renders the prompt as a sequence of chips — one per meaningful phrase — instead of plain text.

• Each chip carries a per-model dot rail underneath it, so you can see at a glance that a phrase was "honored by 2 of 3" models without hovering.
• Hovering a chip previews it on the model columns; clicking pins it (?phrase= in the URL, cleared automatically when you move to a different prompt).
• Chips are colored worst-of: green only if every model that was checked honored the phrase, red with a strikethrough only if all of them missed it, amber for a mixed result. Color is never the only signal — every state also has a glyph and an accessible label.

Attribution uses a deliberate four-state model per (phrase, model) pair:

"Not grounded" is deliberately distinct from "missing": a phrase Augur never checked must never render as a model having failed to paint it.

On the model columns themselves, a phrase overlay draws what the grounder found: a honored phrase gets a bounding box, label, and confidence; an uncertain one gets a plain "low confidence" note; a missing one gets a hatched fail scrim and a "missed" chip. Style-only phrases (that don't map to a region) get a small corner chip instead of a box — Augur never draws a box that isn't backed by data.

This same signal rolls up two other places you'll see while triaging:

• Each matrix cell gets a bottom ribbon with one segment per phrase, so you can read "this model dropped 2 of 5 phrases" straight from the triage matrix without opening the canvas.
• Each model scoreboard card gets a prompt-fidelity meter, shown once grounded data exists for that model.

Related pages

• LLM Evaluations — the shared eval-run model Augur runs plug into.
• Run & Test Case Detail — where Augur runs open from, alongside browser, API, and mobile runs.
• Augur in Studio — the same evidence report and verdicts, embedded in a live Studio workbench surface with in-pane reference approval.