§ Agents & EvalsDocument · REG-EVAL-2026.06

You never trust the model. You trust the evidence that it passed.

Every agent gets an eval suite — quality, consistency, cost, and latency, on cases you choose. Then Regisseur tells you the cheapest model that still passes the bar. For every agent.

Harvard Business Review (June 2026): the strongest agent teams are built from different models. See the evidence ↓

The model comparison matrix — gpt-5.4, Claude Haiku 4.5, and Gemma 3 12B all 100% deterministic, with Gemma starred as the cheapest passing model at $0.00011 per run. — Cheapest model that passesThree models · 100% deterministic · Gemma 3 12B recommended at $0.00011/run

§ 01Eval suites

Test every agent.

Each agent has a Testing tab. Author eval cases — an input plus the expected result or a rubric — or let the platform draft them for you. Run the suite and read four signals, not a single green checkmark.

Quality

Correct output on known inputs

Consistency

Each case sampled repeatedly — a good agent repeats its answer, not a coin-flip

Cost & latency

Tokens, $/run, p50 / p95

Judgment

A calibrated LLM judge scores open-ended output — and is itself calibrated before it can gate anything

An agent's Testing tab showing three eval cases, a judge-calibration panel, and PASS runs. — The Testing tabEval cases, judge calibration, and PASS runs per agent

§ 02Model selection

The cheapest model that still passes.

Run the same suite across every model you'd consider. Regisseur folds the results into a single model × {quality, latency, cost} matrix and stars ★ the cheapest configuration that PASSes — ties broken by lower latency.

Unpriced models show "—", never "$0"; a recommendation is only made among models that actually pass. Premium reasoning where the case is hard, a cheap fast model where it isn't — chosen on evidence, not on vibes.

The model registry in Settings — providers with default and judge badges and per-config pricing. — Model registryDefault and judge models, priced per config

§ 03Drift

Catch drift before your customers do.

Save a baseline. Every later run is rated pass / warn / fail against it across quality, rubric, cost, and latency.

A prompt tweak or model upgrade that quietly degrades quality — or balloons cost — is caught instead of shipping silently. A failing drift check can block the change.

A drift card rating a run against its baseline across deterministic, rubric, cost, and latency — all PASS. — Drift vs baselineDeterministic, rubric, cost, and latency — all PASS

§ 04Why it matters

Different models for different agents.

A regulated vertical can run many specialized agents. Right-sizing each to the cheapest model that still passes — instead of stretching one expensive model across everything — is where quality holds and unit cost falls. It is the concrete mechanism behind the unit-cost half of an AI transformation.

The agent roster — dozens of agents, each with its own model, autonomy ceiling, run count, approval rate, and cost per run. — Right-sized, agent by agentEach agent carries its own model, ceiling, and cost/run

§ Evidence — Harvard Business Review

Harvard Business Review (Mark Purdy, June 18 2026) argues the strongest agent teams are built from different models: when every agent runs on the same model, their errors correlate and blind spots compound. Two studies he cites — diversity-selected agent teams were 25% better at resolving software-engineering problems than agents working individually, and just two diverse agents can “match or exceed the performance of 16 homogeneous agents.”

Regisseur operationalizes model diversity — and goes one step further: it tests every agent across models and assigns the cheapest one that still passes.

HBR · Purdy · June 2026 →

§ Next step

See an eval suite pick the cheapest model that passes.

Bring one agent and a handful of real cases. We'll author the suite, run it across models, and show you the matrix with the recommendation — quality, latency, and cost side by side.

Book a walkthrough →How the pipeline stays deterministic