LABS · STUDY · HARD PROOF

Draft for review

No repro, no bug. No fix, no release.

Hard Proof is a way to accept code that AI agents write, on evidence. A review finding counts only with a repro that a script runs. A release waits until each confirmed bug is closed.

We tested this on our own two products. This page has the result first. The paper has the record in the order of the work.

Puppetmaster Labs · 19 November 2026 · hpflow 0.6.1 · read the paper · PDF · code

Confirmed bugs on one release range8Two AI review tools found 2 each.
Review findings cut before you read them59%124 of 211 findings did not pass two verifiers.
Releases stopped at the gate2 of 64 bugs in new code that an agent wrote.
Cost for one changed function$0.27Review and two verifiers, API-equivalent.
1 · Use it

Audit the change. Fix what is real. Gate the release.

hpf is a command line tool for Python code. It looks only at the functions that changed in a git range, so the cost follows the size of the change.

pip install hpflow

hpf audit --repo . --ref HEAD --changed v1.2.3..HEAD --out ~/audits/next
hpf fix   --out ~/audits/next      # only when the report has confirmed bugs
hpf gate  --out ~/audits/next --ref HEAD
hpf audit

One reviewer agent reads each changed function. A finding counts only when its repro runs. Then two verifier agents on different models must each name a caller that reaches the fault.

hpf fix

An agent writes a fix and a test in a separate worktree. A script check and a judge agent must accept the fix. The tool does not commit: you review each fix.

hpf gate

This step makes no model call. Each confirmed bug must give a new result on the commit that you release, or have a waiver with a reason. If not, the gate fails.

You have a review tool now? Keep it. hpf audit --findings FILE takes the findings of another review tool, or of a person. A worker must show each claim with a repro, and then the same verifiers and the same gate apply. In our test, the 12 findings of two review tools became 11 claims. 7 got a repro, and the verifiers confirmed 3 distinct bugs: the same 3 that a blind judge confirmed.

The audit and the fix step need one agent tool with a login: Claude Code, Codex, or another tool through a command template. They need no other package. The gate needs git and Python only.

2 · The ladder

Each step up needs harder evidence.

An agent that reviews code gives many findings, and most of them are not bugs that a user can reach. The ladder is the filter. These counts are from the audit of 57 functions of our two products, Puppetmaster and Marionette.

  1. 1

    A finding with a repro

    The reviewer writes a small file that calls the function and returns the result. A script runs the file. A finding with no repro that runs is not a finding.

    Decided by a script

    211findings with a repro that runs
  2. 2

    Two verifiers

    Two agents on different models read the code and its callers. Each one must say that the fault is real, and must name a caller that reaches it.

    Decided by two agents that must agree

    87findings passed (59 percent cut). They are 45 distinct bugs
  3. 3

    A fix with a test

    An agent writes the fix and a new test. A script runs each repro again, the new test, and the tests that import the changed module. A judge agent reads the change.

    Decided by a script and one agent

    45fixes that the check and the judge accepted
  4. 4

    A second review

    A second agent session reviews each fix, and runs the new test against the old code. The owner of the code decides each rule that has no exact statement.

    Decided by an agent, CI and the owner

    40fixes stood. 5 were wrong and went to the owner for a rule
  5. 5

    The gate

    Each repro of each confirmed bug runs on the commit that you release. The same result as before means that the bug is open, and the release waits.

    Decided by a script, with no model call

    2of 6 releases stopped

No step of this ladder is a proof. Each grade is the agreement of agents, a script, or CI. The owner of the code decides what is a bug. 51 fixes from this ladder shipped in two releases.

3 · The evidence

What the ladder found in code that we ship.

8 against 2 and 2

One release range, three tools

We gave one release range of Puppetmaster to three tools. The findings of the three went into one list with no tool name, and two verifier agents judged each one.

hpf audit
8
Claude Code review
2
Codex review
2

7 of the 10 confirmed bugs came from hpf only. Each other tool found one bug that hpf missed. Limits: one range and one run for each tool, and the models are not the same. hpf cost about 12 times the Claude Code review.

44 of 95

Known bugs from our own history

We took 95 real bugs from the history of the two products. A finding counts when its repro gives a different result before and after the real fix, so no agent is the judge.

Review, Opus 5.5
44$105
Review, Sonnet 5.5
36$48
Review, no shell
32$45
Full proof pipeline
31$793

The second number is the cost of the run on all 95 bugs. One review pass leaves more than half of the known bugs. So a pass of the gate is not a proof that the code has no bug.

2 of 6

Releases that the gate stopped

The gate now runs before each release of the two products. It stopped two of them. The 4 bugs were in new code that an agent wrote, and some were in our own fixes for earlier findings.

1.51.3
0.9.686
1.51.4
0.9.687
1.52.0
0.9.688

Versions 1.5x are Puppetmaster, and versions 0.9.x are Marionette. Each release went out after the bugs were corrected and the gate passed.

55

Bugs that the tool found in its own release

Before version 0.5.0 we ran hpf on its own release range, and then on each set of fixes. The count went down slowly: a reviewer reads the whole function each time one of its lines changes.

23round 1
10round 2
7round 3
5round 4
6round 5
4round 6
0round 7

7 rounds for $34.54. The last round confirmed nothing. At the release, 52 bugs are closed by their repro and 3 have a waiver.

4 · The cost

$0.27 for each changed function.

The audit of one release range of 20 functions costs $5.38 and takes about 7 minutes. That is the mean of 3 runs, at the API-equivalent price.

First form of the tool
$28.53
Version 0.4: a lean worker
$9.44
Version 0.5: six tools
$5.38

Most of the first cost was not work. Each worker session started with about 83,000 tokens of settings, rules and tool lists of the user of the machine, before it read one line of code. A lean worker does not load that text. Then a fixed list of six tools cut the first request of a worker from 33,500 to 8,800 tokens.

A lower price is no gain if the worker finds less. On 48 known bugs, the worker of version 0.5 detected 18 for $5.25. The worker of version 0.4 detected 17 for $7.24.

For comparison: the full proof pipeline of this study cost $8.34 for one function.

5 · Where a proof belongs

A proof is the second tool, for a small rule that the owner states.

The study started with a different plan: accept code from an agent only with a machine-checked proof. A proof gate shows that the code obeys its laws. It does not show that the laws say enough.

We proved three laws about a lock, and six proof kernels accepted the proofs. Then an adversary agent proved that the same three laws allow a lock that never unlocks.

Design A

The lock we proved

release frees the lock

Lock: free

Law 1 ✓Law 2 ✓Law 3 ✓

Design B

The lock the adversary wrote

release does nothing

Lock: free

Law 1 ✓Law 2 ✓Law 3 ✓

spawn 1acquire 1release 1

After the release, design B still holds the lock. Design B passes the same three laws. Lean checked the two designs and the trace.

We call this result a gap certificate: two designs, a proof that each one passes each law, and a proof that they differ. The owner answers one question for each gap, and the answer becomes a law. The loop stops when an agent proves that no gap is left.

So we use a proof for a small rule where a missed case costs much: a decision table, a permission rule, a price rule, a lock. The command hp of the same package is the proof gate for one Python function.

6 · What did not work

The results that changed our plan.

We started with a proof for each change. The data moved us to a review with receipts for all code, and a proof for the few rules that need one.

$793for 31 of 95 bugs

The full proof pipeline as a bug finder. An agent wrote a model and laws from the documentation of each function, and an adversary looked for a conflict with the code. The pipeline detected 31 of the 95 known bugs. A review detected 44 for $105. The pipeline did find 6 bugs that no review found.

78%of conflicts were misreads

Laws that an agent writes from documentation. We checked the 73 conflicts of the pipeline that still held on the newest releases. In 57 of them the law was wrong and the code was right. None was a bug that a caller reaches. A proof makes a law exact. It does not make a law correct.

3 of 6loops stopped too early

A review loop that runs until it is dry. The loop stops when two reviews in sequence say that the laws are complete. It stopped too early in 3 of 6 law sets, each time on the same fault of a definition. A proof of completeness does not have this fault.

5 of 45fixes were wrong

A fix by an agent, with no second review. 5 fixes passed the script check and the judge, and were wrong: each one changed the result for inputs that the bug does not cover. The second review found them. A human must review each fix before the merge.

3bugs, against 7

One reviewer session for a whole file. It cost $8.94 against $28.53, and it confirmed 3 bugs against 7. More functions in one session get less attention each. It is an option of the tool, and it is not the default.

4 of 48with a small model

A cheaper reviewer. Haiku 5.5 as the reviewer detected 4 of 48 known bugs. Low effort detected 15. The default worker detected 18 of the same 48.

36 of 95in one pass

One pass as the full answer. One review pass detects about a third of the known bugs. Two passes with one set of arguments detected 34 and 36, and 45 together. Each pass finds other bugs, and a second pass costs as much as the first.

12failed workers, one green job

Our own harness. The first CI job of the audit passed with 12 failed workers, because the gate passed with no review report. Our first blind test was not blind: the folder path showed the answer. The paper has each fault with its correction.

7 · Limits

What this page does not show.

8 · Data

Each number on this page, and its source.

A script writes this page from the data files of the study, so each result has one source. The repository has each file, and the command that makes it.

NumberWhat it isPaperData file
8, 2 and 2Confirmed bugs of three tools on one release rangeSection 14study/v2/h2h/result.json
211, 87 and 45Findings with a repro, findings that passed two verifiers, distinct bugsSection 13study/v2/ladder_summary.json
40 of 45Fixes that stood after the second reviewSection 13study/v2/ladder_summary.json
2 of 6Releases that the gate stoppedSection 14study/v2/gate_live.json
44, 36, 32 and 31 of 95Known bugs that each method detectedSection 12study/v2/retro_compare.json
57 of 73Conflicts of the proof pipeline that were a misread of the lawSection 12study/v2/conflict_triage.json
$28.53, $9.44 and $5.38Cost of the audit of one range, in three versions of the toolSection 14study/v2/cost_cut.json, study/v2/release_050.json
18 and 17 of 48Known bugs that the worker of version 0.5 and of version 0.4 detectedSection 14study/v2/release_050.json
55 in 7 roundsBugs that the tool found in its own releaseSection 14study/v2/release_050.json, study/v2/self_audit
7 of 11, and 3Outside claims that got a repro, and the distinct bugs that the verifiers confirmedSection 14study/v2/import_live.json
144, 73 and 66Adversary runs, gap certificates, completeness proofsSection 9labs/data/hard-proof.json (on this site)
24 of 24Correct results in the blind test with generated rulesSection 8labs/data/hard-proof.json (on this site)
12 of 12Reviews with no proof duty that missed one gapSection 10labs/data/hard-proof.json (on this site)
3 of 6Review loops that stopped too earlySection 11labs/data/hard-proof.json (on this site)