The same three models, run through the same 518 tasks, three times each — once bare, once with tools, once with tools plus a governance layer. Governance roughly doubles operational accuracy for every model tested, and the cheapest model wins on cost per correct answer. Every number on this page recomputes from the raw rows linked at the bottom.
STEVE-1 is a governance layer wrapped around frontier models: rules, verification doctrine, approval gates, memory. That's an easy thing to claim and a hard thing to prove. So we built an operational benchmark and ran the same models, on the same tasks, through three different harnesses — changing exactly one variable each time.
No tools, no rules. The model answers from the prompt alone — the floor every "just use a better model" pitch is measured against.
The same model gets file access, execution, search — a competent agent scaffold. No governance corpus, no rules, no approval gates.
Same tools, plus the rules, verification doctrine, and approval-gate discipline STEVE-1 runs under in production.
Same model. Same 518 tasks. The only variable that changes between arms is the harness.
K=1 PRELIMINARY 518 tasks/cell · 9 model×arm cells · 4,662 invocations total
study_v3_checkpoint.jsonl (4,662 rows, sha256 4e47664a47df93db2bd18594dfb96eafe73b40a73c41168404ac9ef2e66cf666) · recomputed 2026-07-16 via report.py · sonnet-vanilla renders 26.8 from the raw recompute (an earlier draft table said 26.9 — a rounding-boundary artifact we caught and corrected, not a re-run).
Governed Sonnet lands inside governed Opus's accuracy range at roughly one-third the cost per correct answer.
| Model | Vanilla | Tooled | Governed by STEVE-1 | $/correct (governed) |
|---|---|---|---|---|
| Opus | 27.5% | 60.1% | 64.8% | $0.064 |
| Sonnet | 26.8% | 51.4% | 63.8% | $0.019 |
| Haiku | 25.5% | 51.1% | 54.9% | $0.014 |
Metric: accuracy = mean(score) over graded rows. $/correct = total cost of all the model's rows (incl. quarantined spend) ÷ count of rows with score ≥ 0.999. See §Metric definitions below.
Every model roughly doubles its raw accuracy once it's governed. The more interesting finding is what happens between models at the governed tier: Sonnet (63.8%) and Opus (64.8%) are one point apart — statistically indistinguishable at this sample size — while Sonnet costs a third of what Opus costs per correct answer. Scaffolding, not model tier, is doing most of the work.
63.7% governed vs. 48.3% tooled — a +15.4pt lift.
McNemar p = 1.149×10⁻¹⁵ · n≈660/cell
−5.6pt delta — below the pre-registered 10-point floor.
Reported as "no measurable difference." Deltas under the floor are noise in both directions, per pre-registration.
95.6% governed vs. 95.2% tooled.
Governance costs nothing on correctness.
Governance is not free. On the governed arm, per task, vs. the tooled arm:
We publish this on the same page as the wins because a benchmark that hides its costs is an ad. The tax is real — and it's repaid many times over in cost per correct answer, where governed Sonnet still beats every vanilla and tooled cell on the table.
study_v3_checkpoint.jsonl, sha256 4e47664a47df93db2bd18594dfb96eafe73b40a73c41168404ac9ef2e66cf666. Metrics recomputed from raw rows 2026-07-16 via report.py.mean(score) over graded rows
Graded = valid AND score present. Rows that failed transport, timed out, or crashed the grader are excluded from the accuracy mean — but never silently.
total cost of ALL the model's rows (incl. quarantined spend) ÷ count of rows with score ≥ 0.999
The denominator only counts passes; the numerator charges every dollar spent, including runs that never produced a gradable answer. Failed spend is not hidden from the cost side.
424 of 4,662 rows (9.1%) are quarantined — transport failures, timeouts, grader crashes. They ship in the downloadable pack with score: null so every exclusion is auditable, not swept away. Nothing is deleted from the dataset; quarantined rows are simply excluded from the accuracy mean while their cost still counts against $/correct.
Vanilla = bare model, no tools, no rules. Tooled = full tool harness, no governance corpus. Governed = tools + the STEVE-1 rules/governance corpus. Same model, same tasks — the only variable is the harness.
Download the full pack — one row per (task, model, arm), 4,662 rows total — and check our math.
Some security-code tasks contain deliberately fake, clearly-labeled credential-shaped bait strings (e.g. sk-ant-FAKE-DO-NOT-USE-…) used to test whether a governed agent refuses to leak them. They are not real keys.
| Field | Meaning |
|---|---|
| task_id | Stable identifier for the task within the frozen battery. |
| category | One of the 10 task categories (e.g. rules-sensitive, security-code). |
| model | haiku / sonnet / opus. |
| arm | vanilla / tooled / governed. |
| rep | Repetition index (k=1 in this pack — always 1). |
| prompt | Exact prompt sent to the model. |
| ground_truth | Sanitized grading spec for the task (programmatic checks, not raw secrets). |
| answer_text | Full model answer text. |
| graded | Boolean — whether this row cleared quarantine and was scored. |
| pass | Boolean — whether the row scored ≥ 0.999. |
| score | Float 0–1, or null for quarantined rows. |
| input_tokens / output_tokens | Token counts for the invocation. |
| cost_usd | Dollar cost of the invocation. |
| wall_s | Wall-clock seconds for the invocation. |
| exit_code / note | Harness-level status for debugging non-graded rows. |
import json
from collections import defaultdict
rows = json.load(open("raw_benchmark_pack.json"))
cells = defaultdict(list)
for r in rows:
cells[(r["model"], r["arm"])].append(r)
for (model, arm), cell in sorted(cells.items()):
graded = [r for r in cell if r["graded"] and r["score"] is not None]
acc = 100 * sum(r["score"] for r in graded) / len(graded)
cost = sum(r["cost_usd"] or 0 for r in cell)
correct = sum(1 for r in cell if r["pass"] is True)
cpc = cost / correct if correct else None
print(model, arm, round(acc, 1), round(cpc, 3) if cpc else "-")