Agent Trust & Assurance · The proof
A benchmark that only reports the win is propaganda. These two publish recall and the misses, precision and the cost — and both improved the product before they proved it.
Benchmark 1 · the structured lane · live, judge-labelled
36 curated queries (12 in-scope · 18 out-of-scope · 6 ambiguous) driven as real turns against a knowledge-graph-scoped agent, with ground truth labelled by the same scope_faithfulness judge the platform uses everywhere — never a benchmark-private prompt. It scores the cheap deterministic trigger's coverage of judge-confirmed drift.
| judge: drifted | judge: faithful | |
|---|---|---|
| trigger fired | TP 2 | FP 9 |
| trigger silent | FN 0 | TN 25 |
Published limits — not footnotes
Precision is 18%, and that is the design. The trigger does recall; the judge does precision. A false positive costs one judge call that clears the answer — never a wrong verdict, never a withheld honest refusal. The number that must be perfect is the false-negative count, and it is.
Recall climbed 0% → 25% → 100% across three iterations — the first run disproved the naïve deterministic-block design (it blocked 10 honest "not in my scope" refusals and missed a prose-only drift), which forced the tiered trigger-plus-judge architecture the platform now ships. The benchmark redesigned the product.
What the run found beyond the gate
36 back-to-back sessions exposed a multi-thread SQLite deadlock that froze the whole server — eliminated with a process-wide serialization layer before the benchmark could even finish. A benchmark hard enough to break things is a benchmark worth publishing.
Benchmark 2 · the knowledge lane · deterministic, offline, zero tokens
18 curated (answer · retrieved-chunks · question) cases — supported answers with benign near-misses, unsupported claims across numbers, verbatim quotes and identifiers, and second-chance cases the first retrieval missed but the corpus supports. Scored by the same production primitive the live gate runs — and because that trigger is deterministic, the whole benchmark reproduces offline with no server, no model, no credentials.
Published limits — not footnotes
The one false negative is deliberate. Case uns-id-01 fails because a single novel identifier never triggers alone — the ≥3-identifier floor is the false-positive discipline that keeps the 0% column at 0%. The miss is published by case id and pinned in CI rather than tuned away.
"Unsupported" never means "false." The trigger checks support in the retrieved corpus — after a targeted second-chance re-search — and says exactly that, on the card and in the answer itself.
What the first run found
A supported quote ending in a period inside the quotation marks read as unsupported — trailing-punctuation brittleness in the quote matcher. Fixed in the production primitive; the failing case stays in the dataset as the regression. The benchmark improved the product on day one.
Method
Live + judge-labelled (structured lane)
Ground truth needs semantic judgement ("did this answer drift?"), so an LLM judge labels it — the same judge the platform ships. Honest, but reproducing it needs a running server, credentials, and ~$0.03.
Deterministic + offline (knowledge lane)
"Is this claim in these chunks?" is decidable in code, so the benchmark is the production primitive on a committed dataset. Zero tokens, zero infrastructure — the published numbers are pinned in CI, and any change that moves them fails the build.
Every enforcement claim in these stories traces to one of these two benchmarks or to a named CI suite — 1,092 assertions across 24 suites at last full run. If a number on this page moves, a test fails first.