AI Code Review Needs Eval Provenance for Agent-Run Benchmarks
Public benchmark scores for coding agents keep climbing, but most engineering teams still cannot answer the question that actually matters: what happened during the run that produced that score? For AI code review, this gap is dangerous. If you cannot replay the repo state, inspect the tool trace, and verify the grading logic, then the benchmark result is closer to marketing than operational evidence.
Key Takeaways
- Agent benchmark scores are only useful when the run is reproducible and reviewable.
- Eval provenance should capture repo snapshot, prompt and policy versions, tool traces, validation commands, and grading rules.
- Without provenance, teams cannot tell whether a benchmark win translates to safer pull request review.
- Provenance gives engineering leaders a clean path from public benchmark hype to internal rollout decisions.
- Propel teams can treat eval provenance as a first-class review artifact, not an optional lab note.
TL;DR
In 2026, benchmark scores for coding agents move too fast to trust on their own. Require eval provenance for every agent-run benchmark or internal experiment: repo snapshot, tool scope, prompt version, grader definition, and replayable validation steps. That is how you decide whether a model should review production pull requests instead of just winning a leaderboard screenshot.
Why this topic is spiking right now
On March 10, 2026, METR published evidence that many pull requests capable of passing SWE-bench would still not be merge-ready for maintainers in the real world. On March 19, 2026, TLDR Dev summarized the mood with the blunt headline that AI coding is gambling. Around the same time, Latent.Space argued that the market is moving beyond SWE-bench Verified alone, and Interconnects framed the moment as part of a post-benchmark era.
The product takeaway is not that benchmarks are useless. It is that benchmarks now need provenance. If a model or agent achieved a strong result, reviewers need to know which repo state it saw, which tools it touched, what constraints were in force, and how the run was graded before that result can influence production routing.
- METR, March 10, 2026: Many SWE-bench-Passing PRs Would Not Be Merged
- TLDR Dev, March 19, 2026: AI coding is gambling
- The Pragmatic Engineer: Building Claude Code with Boris Cherny
- Latent.Space: The End of SWE-Bench Verified
- Interconnects: Opus 4.6, Codex 5.3, and the post-benchmark era
What eval provenance actually means
Eval provenance is the minimum evidence needed to explain how an agent benchmark result was produced. It is not chain-of-thought capture, and it is not a raw log dump that nobody will read. It is a compact, structured record that lets another engineer reproduce the setup and audit the conclusion.
This is the benchmark counterpart to the session artifacts we recommend in AI code review session provenance. If session provenance explains how a pull request was authored, eval provenance explains how a benchmark or model comparison was run.
Why benchmark scores without provenance fail engineering teams
Teams rarely deploy a new review model because it beat one benchmark by two points. They deploy when they believe the benchmark result predicts lower review noise, better severe issue capture, and safer merge decisions in their own repositories. Provenance is what lets them bridge that gap.
| Without provenance | With provenance | Why it matters |
|---|---|---|
| One score and one chart | Replayable run artifact pack | Reviewers can verify what the agent really saw and did |
| Unknown repo snapshot | Commit hash, dependency lock, fixture set | Results stay stable across reruns and tool upgrades |
| Opaque prompt and policy stack | Versioned prompt, harness, and guardrail IDs | Teams can separate model gains from prompt engineering drift |
| Black-box grading | Explicit grader, judge, thresholds, and failure modes | A benchmark win becomes audit-ready instead of arguable |
The minimum provenance fields every agent benchmark needs
Most teams do not need an expensive eval platform to start. They need a schema and the discipline to fill it in every time. A practical starting point looks like this:
Minimum schema
- Run header: task set, run ID, timestamp, model ID, provider, temperature, seed.
- Repo state: base commit, fixture branch, dependency lockfile hash, test dataset version.
- Execution policy: prompt version, tool permissions, sandbox mode, max step budget.
- Tool trace: commands executed, files opened or edited, external services touched.
- Validation trace: test commands, lint runs, failed retries, human overrides.
- Grading trace: scorer version, pass criteria, judge model if used, manual spot checks.
- Outcome trace: final score, cost, latency, token usage, unresolved failure reasons.
A compact artifact format reviewers can actually use
Provenance fails when it becomes a giant transcript nobody wants to inspect. Keep the human surface small and structured. The full logs can be retained for incident response, but the default artifact should be short enough to review in minutes.
{
"runId": "eval-2026-03-19-0142",
"taskSet": "pr-review-high-risk-v3",
"model": "provider/model-version",
"repo": {
"commit": "a1b2c3d",
"lockfileSha256": "..."
},
"policy": {
"promptVersion": "review-policy-v12",
"sandbox": "workspace-write",
"maxSteps": 40
},
"validation": {
"commands": ["pnpm test --filter review-evals"],
"failures": []
},
"grading": {
"rubricVersion": "severity-precision-v4",
"judge": "human+rule"
},
"result": {
"score": 0.82,
"costUsd": 4.63,
"latencySec": 91
}
}This structure aligns naturally with the artifact thinking in evidence-first AI code review. The difference is that the subject is the eval run itself rather than the pull request under review.
How this changes model selection for AI code review
Once provenance is available, teams can stop debating benchmark screenshots and start comparing operational facts. Was the run dependent on broad tool access? Did the agent silently retry until it found a passing path? Did the grading rubric overweight shallow comments? Did cost explode on the hard cases you actually care about?
That is the missing link between public benchmark signals and the internal evaluation loop we laid out in post-benchmark AI code review evals. Provenance lets you reuse outside benchmark news as a hypothesis while still making a sober rollout decision inside your own workflow.
Use provenance to separate model quality from harness quality
A lot of benchmark movement in 2026 does not come from a raw model jump alone. It comes from better harness design, richer tool use, smarter retry logic, and more selective grading. Those improvements matter, but they should be visible. Otherwise teams may think they are buying one kind of capability while actually inheriting a much more fragile stack.
This is especially important when reviewing agent-authored code. If a benchmark-leading model depends on a permissive harness that your production environment would never allow, then the benchmark win is not portable to your review system. Provenance makes that mismatch obvious.
30-day rollout plan for engineering teams
Suggested rollout
- Week 1: define the provenance schema and make it mandatory for all new eval runs.
- Week 2: backfill provenance for your top 10 benchmark claims and current model routes.
- Week 3: add provenance checks to review model upgrade proposals and benchmark reports.
- Week 4: require provenance before any model can influence Tier 2 or Tier 3 PR routing.
Pair this rollout with the controls in agentic engineering code review guardrails so benchmark evidence and production policy evolve together.
How Propel helps
Propel gives engineering teams a way to connect model evaluation to real review operations. That means keeping artifact quality high, routing risky changes with policy awareness, and making it clear why a model or agent earned trust in the first place. Benchmarks can start the conversation, but provenance is what makes the decision defensible.
FAQ
Is eval provenance only useful for benchmark teams?
No. It is useful anywhere a benchmark or internal experiment influences production model routing, review policy, or purchasing decisions.
Do we need to store full agent transcripts?
Usually no. Store a compact provenance artifact by default and keep full logs only when you need incident forensics, regulated retention, or deep debugging.
What is the first field most teams forget?
The grader definition. Teams often record the model and prompt but forget to version the scoring logic that turned the run into a headline number.
How does this connect to code review specifically?
Review systems make merge-affecting decisions. If the eval evidence behind those decisions is not reproducible, your rollout is weaker than the policy boundary it is supposed to protect.
Turn benchmark excitement into reviewable rollout decisions
Propel helps teams connect benchmark results to evidence, risk routing, and production code review policies before a model ever touches a high-impact PR.
