LABS · STUDY · HARD PROOF
Draft for review
No repro, no bug. No fix, no release.
Hard Proof is a way to accept code that AI agents write, on evidence. A review finding counts only with a repro that a script runs. A release waits until each confirmed bug is closed.
We tested this on our own two products. This page has the result first. The paper has the record in the order of the work.
Audit the change. Fix what is real. Gate the release.
hpf is a command line tool for Python code. It looks only at the functions that changed in a git range, so the cost follows the size of the change.
pip install hpflow hpf audit --repo . --ref HEAD --changed v1.2.3..HEAD --out ~/audits/next hpf fix --out ~/audits/next # only when the report has confirmed bugs hpf gate --out ~/audits/next --ref HEAD
One reviewer agent reads each changed function. A finding counts only when its repro runs. Then two verifier agents on different models must each name a caller that reaches the fault.
An agent writes a fix and a test in a separate worktree. A script check and a judge agent must accept the fix. The tool does not commit: you review each fix.
This step makes no model call. Each confirmed bug must give a new result on the commit that you release, or have a waiver with a reason. If not, the gate fails.
You have a review tool now? Keep it. hpf audit --findings FILE takes the findings of another review tool, or of a person. A worker must show each claim with a repro, and then the same verifiers and the same gate apply. In our test, the 12 findings of two review tools became 11 claims. 7 got a repro, and the verifiers confirmed 3 distinct bugs: the same 3 that a blind judge confirmed.
The audit and the fix step need one agent tool with a login: Claude Code, Codex, or another tool through a command template. They need no other package. The gate needs git and Python only.
Each step up needs harder evidence.
An agent that reviews code gives many findings, and most of them are not bugs that a user can reach. The ladder is the filter. These counts are from the audit of 57 functions of our two products, Puppetmaster and Marionette.
- 1
A finding with a repro
The reviewer writes a small file that calls the function and returns the result. A script runs the file. A finding with no repro that runs is not a finding.
Decided by a script
211findings with a repro that runs - 2
Two verifiers
Two agents on different models read the code and its callers. Each one must say that the fault is real, and must name a caller that reaches it.
Decided by two agents that must agree
87findings passed (59 percent cut). They are 45 distinct bugs - 3
A fix with a test
An agent writes the fix and a new test. A script runs each repro again, the new test, and the tests that import the changed module. A judge agent reads the change.
Decided by a script and one agent
45fixes that the check and the judge accepted - 4
A second review
A second agent session reviews each fix, and runs the new test against the old code. The owner of the code decides each rule that has no exact statement.
Decided by an agent, CI and the owner
40fixes stood. 5 were wrong and went to the owner for a rule - 5
The gate
Each repro of each confirmed bug runs on the commit that you release. The same result as before means that the bug is open, and the release waits.
Decided by a script, with no model call
2of 6 releases stopped
No step of this ladder is a proof. Each grade is the agreement of agents, a script, or CI. The owner of the code decides what is a bug. 51 fixes from this ladder shipped in two releases.
What the ladder found in code that we ship.
One release range, three tools
We gave one release range of Puppetmaster to three tools. The findings of the three went into one list with no tool name, and two verifier agents judged each one.
7 of the 10 confirmed bugs came from hpf only. Each other tool found one bug that hpf missed. Limits: one range and one run for each tool, and the models are not the same. hpf cost about 12 times the Claude Code review.
Known bugs from our own history
We took 95 real bugs from the history of the two products. A finding counts when its repro gives a different result before and after the real fix, so no agent is the judge.
The second number is the cost of the run on all 95 bugs. One review pass leaves more than half of the known bugs. So a pass of the gate is not a proof that the code has no bug.
Releases that the gate stopped
The gate now runs before each release of the two products. It stopped two of them. The 4 bugs were in new code that an agent wrote, and some were in our own fixes for earlier findings.
Versions 1.5x are Puppetmaster, and versions 0.9.x are Marionette. Each release went out after the bugs were corrected and the gate passed.
Bugs that the tool found in its own release
Before version 0.5.0 we ran hpf on its own release range, and then on each set of fixes. The count went down slowly: a reviewer reads the whole function each time one of its lines changes.
7 rounds for $34.54. The last round confirmed nothing. At the release, 52 bugs are closed by their repro and 3 have a waiver.
$0.27 for each changed function.
The audit of one release range of 20 functions costs $5.38 and takes about 7 minutes. That is the mean of 3 runs, at the API-equivalent price.
Most of the first cost was not work. Each worker session started with about 83,000 tokens of settings, rules and tool lists of the user of the machine, before it read one line of code. A lean worker does not load that text. Then a fixed list of six tools cut the first request of a worker from 33,500 to 8,800 tokens.
A lower price is no gain if the worker finds less. On 48 known bugs, the worker of version 0.5 detected 18 for $5.25. The worker of version 0.4 detected 17 for $7.24.
For comparison: the full proof pipeline of this study cost $8.34 for one function.
A proof is the second tool, for a small rule that the owner states.
The study started with a different plan: accept code from an agent only with a machine-checked proof. A proof gate shows that the code obeys its laws. It does not show that the laws say enough.
We proved three laws about a lock, and six proof kernels accepted the proofs. Then an adversary agent proved that the same three laws allow a lock that never unlocks.
The lock we proved
release frees the lock
Lock: free
Law 1 ✓Law 2 ✓Law 3 ✓
The lock the adversary wrote
release does nothing
Lock: free
Law 1 ✓Law 2 ✓Law 3 ✓
After the release, design B still holds the lock. Design B passes the same three laws. Lean checked the two designs and the trace.
We call this result a gap certificate: two designs, a proof that each one passes each law, and a proof that they differ. The owner answers one question for each gap, and the answer becomes a law. The loop stops when an agent proves that no gap is left.
- 144 adversary runs gave 73 gap certificates and 66 completeness proofs. 5 runs gave no accepted result.
- In a blind test with generated rules, the adversaries gave the correct result in 24 of 24 runs, up to 512 laws.
- A review with no proof duty missed one gap in 12 of 12 reviews. The proof adversary found that gap 3 of 3 times.
- A test links the model to the real code. For the lock it ran 1.8 million events against the real lock class, with 0 differences.
- On a product function, a decision table with 4 inputs got 5 laws. Lean accepted a proof that the code passes them, and a proof that the laws leave none of the 16 cases open. The work took about ten minutes with no worker agent.
So we use a proof for a small rule where a missed case costs much: a decision table, a permission rule, a price rule, a lock. The command hp of the same package is the proof gate for one Python function.
The results that changed our plan.
We started with a proof for each change. The data moved us to a review with receipts for all code, and a proof for the few rules that need one.
The full proof pipeline as a bug finder. An agent wrote a model and laws from the documentation of each function, and an adversary looked for a conflict with the code. The pipeline detected 31 of the 95 known bugs. A review detected 44 for $105. The pipeline did find 6 bugs that no review found.
Laws that an agent writes from documentation. We checked the 73 conflicts of the pipeline that still held on the newest releases. In 57 of them the law was wrong and the code was right. None was a bug that a caller reaches. A proof makes a law exact. It does not make a law correct.
A review loop that runs until it is dry. The loop stops when two reviews in sequence say that the laws are complete. It stopped too early in 3 of 6 law sets, each time on the same fault of a definition. A proof of completeness does not have this fault.
A fix by an agent, with no second review. 5 fixes passed the script check and the judge, and were wrong: each one changed the result for inputs that the bug does not cover. The second review found them. A human must review each fix before the merge.
One reviewer session for a whole file. It cost $8.94 against $28.53, and it confirmed 3 bugs against 7. More functions in one session get less attention each. It is an option of the tool, and it is not the default.
A cheaper reviewer. Haiku 5.5 as the reviewer detected 4 of 48 known bugs. Low effort detected 15. The default worker detected 18 of the same 48.
One pass as the full answer. One review pass detects about a third of the known bugs. Two passes with one set of arguments detected 34 and 36, and 45 together. Each pass finds other bugs, and a second pass costs as much as the first.
Our own harness. The first CI job of the audit passed with 12 failed workers, because the gate passed with no review report. Our first blind test was not blind: the folder path showed the answer. The paper has each fault with its correction.
What this page does not show.
- A pass of the gate says that each bug that two verifier agents confirmed is closed or waived. It does not say that the code has no bug.
- The tools work on Python functions only.
- We tested the tools on our own products. Most runs used Claude models through Claude Code.
- The comparison with two review tools is one release range and one run for each tool.
- We made no agent run on Windows. The fix step has a live run on Claude Code only.
- The proof results are on small domains that we selected. A stand-in gave the answers of the human in the proof loop.
- The parts of the proof gate are known work: the checker is the comparator of Lean, and hardware verification has the completeness criterion. We do not claim that no earlier system has this loop.
- Each cost is the API-equivalent price that the provider tool reported.
Each number on this page, and its source.
A script writes this page from the data files of the study, so each result has one source. The repository has each file, and the command that makes it.
| Number | What it is | Paper | Data file |
|---|---|---|---|
| 8, 2 and 2 | Confirmed bugs of three tools on one release range | Section 14 | study/v2/h2h/result.json |
| 211, 87 and 45 | Findings with a repro, findings that passed two verifiers, distinct bugs | Section 13 | study/v2/ladder_summary.json |
| 40 of 45 | Fixes that stood after the second review | Section 13 | study/v2/ladder_summary.json |
| 2 of 6 | Releases that the gate stopped | Section 14 | study/v2/gate_live.json |
| 44, 36, 32 and 31 of 95 | Known bugs that each method detected | Section 12 | study/v2/retro_compare.json |
| 57 of 73 | Conflicts of the proof pipeline that were a misread of the law | Section 12 | study/v2/conflict_triage.json |
| $28.53, $9.44 and $5.38 | Cost of the audit of one range, in three versions of the tool | Section 14 | study/v2/cost_cut.json, study/v2/release_050.json |
| 18 and 17 of 48 | Known bugs that the worker of version 0.5 and of version 0.4 detected | Section 14 | study/v2/release_050.json |
| 55 in 7 rounds | Bugs that the tool found in its own release | Section 14 | study/v2/release_050.json, study/v2/self_audit |
| 7 of 11, and 3 | Outside claims that got a repro, and the distinct bugs that the verifiers confirmed | Section 14 | study/v2/import_live.json |
| 144, 73 and 66 | Adversary runs, gap certificates, completeness proofs | Section 9 | labs/data/hard-proof.json (on this site) |
| 24 of 24 | Correct results in the blind test with generated rules | Section 8 | labs/data/hard-proof.json (on this site) |
| 12 of 12 | Reviews with no proof duty that missed one gap | Section 10 | labs/data/hard-proof.json (on this site) |
| 3 of 6 | Review loops that stopped too early | Section 11 | labs/data/hard-proof.json (on this site) |
