LABS · PAPER · PROOF GATES FOR AGENT CODE

Draft for review

Proved is not finished: completeness checks for proof gates on agent-written code

Cary Palmer · 18 November 2026 · Lean 4.36.0 · Puppetmaster 1.51.0 · visual summary

Abstract

A proof gate accepts an agent's code change only when a deterministic checker accepts a machine-checked proof for each law that a human owns. The gate forces "the code obeys the laws". It does not force "the laws say enough". We show that this second gap is real, that agents can find it with proofs, and that it can be closed with a proof.

We proved three laws for a model of a lock, and six proof kernels accepted the proofs. Adversary agents then got the laws and one task: find two designs that pass each law and behave differently. In 16 of 16 runs on three model tiers, the agent supplied two lock designs, proofs that each design passes each law, and a proved trace on which they differ. One design never releases the lock. We call this object a gap certificate. The same checker that accepts a solution accepts it.

A gap certificate becomes one concrete question for the human. The answer becomes a new law. With the new law, the agents got a second task: find a gap or prove that none exists. In 16 of 16 runs the agent proved that two designs that pass each law behave the same on each sequence of events. The folder had no example of such a proof. The loop stops when the checker accepts that theorem.

We repeated the loop on real shipped code: the command puppetmaster models sync, with one law for each sentence of its release notes. Adversary agents proved that the release notes allow a sync that writes the new rows in a different order, and a sync that writes each new row two times. Link tests compare the two Lean models with the real Python code on 2.4 million events and cases, with 0 differences, and the lock test fails on the real release that had the bug.

In a third domain no pilot text enters the laws. An agent wrote the laws for the lock's stale-recovery rule from the English documentation of the code. Adversary agents found the gaps, the answer to each question was the behavior of the shipped code, and a law-author agent wrote a general law for each answer. The loop stopped on a completeness proof for each of three law sets. The number of questions was 2 for the laws by Haiku 5.5, 1 for the laws by Sonnet 5.5, 2 for the laws by Opus 5.5.

We ran the same test on the sync command. The law authors got seven sentences of its documentation, and a model file that also holds a mirror of the code. Six law sets by three models each left the same behavior open: the order of the new rows, which the documentation does not state. Each loop ended on a completeness proof after one to three questions, for $1.43 to $3.60. The adversaries also found a fault in a definition that the pilot owned: a "legal" input that was not intended to be legal.

A scale test with generated rules did not find the limit of the loop. With the laws in random order and no reference rule, adversaries gave the correct result in 24 of 24 runs, up to 512 laws over 262,144 cases, for $0.16 to $1.05 for one run. The folder path of each of these runs told the worker which kind of result was true, so a second set ran in neutral folders (Section 10). They wrote their own programs to build the proofs, and the gate checked the proofs. The one proof that the workers did not finish in this study is the completeness proof for the lock, in two attempts by a weaker model from a second vendor.

A last test asks if the proof is necessary. A reviewer agent with no proof duty got the same question at 48 decision points of the study. It found 22 of 27 gaps, at about the same cost and in less time. It missed one gap in 12 of 12 reviews, with two models, with and without tools: a legal input that a definition permits and its name does not suggest. The proof adversary found that gap at each of the three points. A wrong review had the same form as a correct review.

Two tests look at the loop as a whole. A review loop that runs until two reviews in sequence say "complete" ended early in 3 of 6 law sets, each time on that same definition fault. Where it was correct, it cost the same as the proof loop, but it gives no evidence that it is done. As one durable flow, in which a later round resumes the session of its worker, a later round of the proof loop cost $0.06 to $0.31, against $0.40 to $0.49 for a first visit. On 95 real bugs of our own code, a review that is pointed at the function detected 44, and a full proof pipeline detected 31 for about 8 times the cost. The pipeline detected 6 of the 40 bugs that three review arms missed. The other findings of those reviews went up a ladder of evidence: a repro on the newest code, two independent verifiers, a fix with a test, and a second review. 45 distinct bugs in the two products were confirmed, and 51 fixes shipped in two releases after the owner decided the rules that had no precise statement. One of those rules has a completeness proof now.

The completeness criterion is from hardware verification, and distinguishing inputs and mutation tests for specifications are known. The contribution is a composition of these ideas into a loop for agents in Lean, with an artifact for each run: proof-carrying gap certificates from an untrusted adversary, one question for the human for each gap, and a completeness theorem as the stop condition.

Gap certificates accepted73
Completeness proofs accepted66
Link test, differences0
Lowest cost of one gap certificate$0.17

A machine-checked proof that code obeys its laws is not evidence that the laws are finished. An agent can prove what the laws leave open, and a theorem can show when no difference is left for a chosen observation in a chosen model.

This is a feasibility result. It covers three small domains that the pilot selected, one family of generated rules and one checker. Completeness here means that the laws fix the observation inside the model. It does not mean that the laws say what the human wants, or that the code is correct. Section 17 lists the limits.

Revision 12, 19 November 2026. Section 14 has version 0.5 of the tool (no orchestrator, any agent tool, a lower cost, the audit of the tool on its own release, one comparison with two review tools) and version 0.6 (the findings of another reviewer as input). Sections 10 to 14 are new since the first revision: a review with no proof duty, two loop tests, real bugs of our own code, an audit that takes review findings to fixes, and our recommendation with the tools. Revisions 4 and 5 have the full run of the proof pipeline on the real bugs and the audit of its conflict certificates (Section 12), the owner's decisions, the shipped fixes and the first proof on a product function (Section 13). Section 10 also corrects the blind set of Section 8: the first revision said that the adversary did not know which kind of result was true, and the folder path showed it.

  1. The problem
  2. Definitions
  3. Setup
  4. Domain 1: a lock
  5. Domain 2: shipped code
  6. Domain 3: laws written by an agent
  7. Domain 2 again: laws written by an agent
  8. A scale test on generated rules
  9. Results
  10. A review with no proof duty
  11. The loop, a review loop, and resumed sessions
  12. Real bugs of our own code
  13. From findings to fixes: an audit with grades of evidence
  14. What we recommend, and how to use the tools
  15. What the human does
  16. Prior work
  17. Limits and disclosures
  18. Data and reproduction

1. The problem

An agent writes code faster than a human can read it. A review by a human or by a second model is a judgment. Tests show that some inputs work. A proof gate replaces the judgment with a check: the agent must supply a proof that the code obeys a formal law, and a small trusted checker accepts or rejects the proof.

The known weak point is the law. A proof shows that the code matches the law. It does not show that the law matches what the human wants. The standard answer is that a human reads the laws. Our first experiments show why that answer is not enough.

We wrote three laws for a lock and proved them. The author of the laws did not see that no law says "a release frees the lock". The three laws allow a lock that never unlocks. A reader of three short Lean statements has the same blind spot. The question of this paper is narrow:

Can the gap between "proved" and "finished" be measured and closed with the same machinery that checks the proofs?

2. Definitions

The gate

A law is a formal statement about a design. A human owns the laws and each definition that a law mentions. The agent owns the design and the proofs. The gate is a deterministic function from the laws, the design and the proofs to accept or reject. It makes no model call. A law with no proof gives a reject.

The gate has three parts. A hash lock rejects a run when a human-owned file changes. A structure check requires a proof for each law and a "kill" proof for each known-bad design. The proof check is Lean's lake comparator: it compares each statement with a challenge file that the human owns, audits the axioms, and replays the proof in the kernel. With one flag it also runs five more kernels.

The gap certificate

Fix an observation: the part of the behavior that a user of the system can see. A gap certificate for a law set has four parts:

  1. Two designs, A and B.
  2. A proof that A passes each law, and a proof that B passes each law.
  3. One legal input.
  4. A proof that the observations of A and B differ on that input.

A gap certificate is an objection to the laws that has the same standard as a solution. An untrusted agent can write it, because the checker accepts it only when each part is proved. Inside the model it cannot be a false alarm: the two designs do pass each law, and they do differ. It can still point at a fault outside the laws, for example a definition of a legal input that is wrong (Section 7).

Complete laws

A law set is complete for an observation when each two designs that pass each law give the same observation on each legal input:

theorem laws_complete :
  ∀ (I J : Design), Passes I → Passes J →
    ∀ input, observe I input = observe J input

A gap certificate and a completeness proof exclude each other. For a law set that some design passes, one of the two is true, and a proof of one of them answers the question. This criterion is not ours. Hardware verification uses it for sets of properties (Section 16).

The loop

  1. Prove. An agent proves that the design obeys each law. The gate checks.
  2. Attack. A second agent must supply a gap certificate or a completeness proof. The gate checks.
  3. Ask. For a gap, the gate prints the input and the two observations. The human selects one.
  4. Add. The answer becomes a law. Go to step 1.
  5. Stop. The loop ends when the gate accepts a completeness proof.

At the stop, the laws leave no behavior open for the chosen observation. The human made each choice, and each choice was a concrete case.

The stop is a certificate, not a guarantee. The loop does not end in general. A domain with unbounded behavior can give a new question in each round, and a true completeness statement can be too hard for the prover. The loops in this study had a limit of six or eight rounds and ended after two to four.

3. Setup

4. Domain 1: a lock

The model

The world is an operating system that gives a pid to a new process after the old process stops. Each process start gets a new start token. A lock design selects what the lock stores about its owner, and how the lock decides that the owner is alive. Four events exist: spawn, exit, acquire, release. The observation is the set of processes in the critical section. The lock cannot read that set.

The model stands for a real component: puppetmaster.interprocess_lock.InterProcessFileLock, which stores a pid and a kernel start token in a lock file.

The first law was weak

Our plan had one law: two processes are never in the critical section together. The known bug for this component is pid reuse: the lock stores the pid alone, the owner dies, and a new process gets the pid. Lean proves that this buggy design obeys the planned law (mutex_alone_misses_pid_only). Pid reuse does not put two processes inside. It makes a dead owner look alive, so the lock stays shut.

Law 2 catches it: when the lock refuses a request, a live process is in the critical section. Lean proves that law 2 is false for the pid-only design, with a trace of four events.

The gate against cheats

We ran eleven scripted cheat attempts against the gate.

Cheat attemptVerdictWhat gave the verdict
sorry in a proofRejectComparator: illegal axiom sorryAx
New axiomRejectComparator: illegal axiom
native_decideRejectComparator: illegal axiom
Prove a different statementRejectComparator: statement mismatch
Define a new law with the same nameRejectComparator: statement mismatch
Edit the law fileRejectHash lock
Edit the challenge fileRejectHash lock
Remove the mutant kill from the configRejectHash lock
Ship the known-bad design as the codeRejectBuild: the proof does not check
Ship a lock that always grantsRejectBuild: the proof does not check
Edit the challenge and sign againAcceptLimit of a lock file in the worker's folder. With the pins in the flow graph: reject.

The last row is a limit of a gate whose lock file is in the worker's folder. From experiment 3 on, the hash of the lock file and of the gate script is pinned in the flow graph. Scripted runs of "edit the challenge and sign again" and "change the gate to exit 0" then give a reject.

The link to real code

A proof about a model says nothing about code until a test links them. The link test sends the same random event traces to the Lean model and to the real lock class on a real lock file. After each event, the owner record and the critical-section set must agree. The kernel answers for "which pid am I", "what is the start token of this pid" and "is this pid alive" are simulated, because a test cannot make the kernel use a pid again.

MeasureValue
Code under testInterProcessFileLock, Puppetmaster 1.50.0
Traces60,015 (three seeds)
Events1,800,081
Traces where the code and the model differ0
Traces where the code breaks a law0
Requests made while a new process had the owner pid13,118
Changed code: pid_reused always answers Falsediffers in 413 of 2,005 traces
Changed code: the kernel gives no start tokendiffers in 413 of 2,005 traces

The same test ran against five real releases. Release 1.28.6 is the last release before the pid-reuse fix.

ReleaseTracesTraces that differNote
v1.28.65,0052,518last release before the pid-reuse fix
v1.29.05,0050the release with the fix
v1.40.05,0050
v1.50.05,0050
v1.51.05,0050

The first trace that differs on 1.28.6 is the trace that Lean used to prove law 2 false for the pid-only design, with one more process.

An impossible challenge

We added one requirement: the lock record must not contain the start token. Lean proves that no lock design can satisfy law 1, law 2 and this requirement together (challenge_is_impossible). A lock with no token cannot tell "the owner died and a new process has its pid" from "that new process took the lock". Law 2 says grant in the first state, and law 1 says refuse in the second.

Four workers got this challenge. One had a neutral brief. Three had a brief with pressure ("this change must ship today") and write access to each file. Each worker changed no file and reported the conflict with the correct reason. One Haiku 5.5 worker with a shell also wrote and compiled its own proof of the impossibility.

The gap

The adversary brief did not say where a gap can be. It did say that the task was to find one, and the folder had the proved lock design and a pinned example in which a release takes a process out of the critical section (Section 17). Each accepted certificate has the same content: design B is the proved lock with a release that does nothing. On the trace spawn, acquire, release, the process is out of the critical section for design A and in it for design B. Design B passes law 1, law 2 and law 3.

The question for the human is one line: after its own release, is the process out, or in? We answered "out". This answer is law 4.

Completeness

With four laws, each worker decided that no gap is left and proved it. The proof shows that the critical-section set after each event is a function of the operating-system state and the set before the event. The function does not name a design. Laws 1, 2 and 4 are sufficient for this result. Law 3 adds no observable constraint.

5. Domain 2: shipped code

The code and its laws

puppetmaster models sync adds the starter models that a model registry lacks. It shipped in Puppetmaster 1.51.0 on the day of this study. The real function is puppetmaster.registry_sync.plan_starter_sync.

The Lean model refSync mirrors the Python code line for line. We wrote seven laws, one for each sentence of the release notes: the old rows stay as they are, only absent starter entries are added, no absent entry is left, a new entry is disabled, an entry on a locked adapter stays disabled, the enable flag keeps the starter state, and a second run changes nothing.

Proofs with no examples

The folder for this domain had no proofs. Each worker proved the seven laws for refSync. The gate of the flow accepted each result on the first check, with no repair visit.

The link

The link test sends random registries, starter lists and settings to refSync and to the real Python function. One part is replaced: the starter list of the case stands in for the built-in starter list.

MeasureValue
Code under testplan_starter_sync, Puppetmaster 1.51.0
Random cases600,000 (three seeds)
Cases where the code and the model differ0
Rows added in those cases1,386,883
Changed code: the enable flag ignores the adapter lockdiffers in 2,741 of 20,000 cases
Changed code: no identity checkdiffers in 6,720 of 20,000 cases

The gaps

Each adversary found a gap in the seven laws. Two kinds came back:

The shipped code writes each row one time, in starter order. The gap is in the written description of the feature, not in the code. A second implementation that follows the release notes can be wrong in these two ways and still pass.

The answer to the two questions is the behavior of the shipped code. It became law 8: the new rows are the absent starter entries, one time each, in starter order.

Completeness

With eight laws, each worker proved that two sync designs that pass each law give the same registry for each legal input.

6. Domain 3: laws written by an agent

In domains 1 and 2 the pilot wrote the laws, and the pilot knew each gap before the adversary ran. A reader can say that the laws were made to have a gap. Domain 3 removes the pilot from the laws.

The rule and its description

The real lock removes a lock file that a different process left, but only in some cases. The code is InterProcessFileLock._recover_stale_owner. Its documentation has three sentences about this rule. One of them: "A later process recovers an orphan whose owner PID is no longer alive; malformed/unknown owners are recovered after stale_after seconds."

The pilot wrote the model types, a mirror of the code (ref), and a link test. The link test compares ref with the real method on a full grid of 144 cases, on real lock files. The two agree on each case. Two changed versions of the code differ in 18 cases each.

Three agents write the laws

Three workers (Haiku 5.5, Sonnet 5.5, Opus 5.5) each got the three sentences and one task: write one law for each stated requirement, and no law for a case that the description does not state. The gate accepted a law set when the mirror of the shipped code passes each law, some rule fails a law, and no law names the mirror. Each worker wrote six laws. The pilot read the laws after the gate accepted them, and did not edit them or prove a result about them before the next step.

The loop with no pilot text

For each law set, a script ran the loop with Sonnet 5.5 workers. An adversary supplied a gap certificate or a completeness proof. For a gap, the answer to the question was the behavior of the shipped code for that case. A law-author worker then added a general law for that answer. The gate checked that each old law was unchanged and that the mirror still passed.

Laws written byLaws at the startQuestionsLaws at the endResultLoop timeLoop cost
Haiku 5.5628completeness proof114 s$1.99
Sonnet 5.5617completeness proof80 s$1.26
Opus 5.5628completeness proof152 s$2.12

These are the questions that the gate printed, with the answer and the name of the law that an agent then added:

Laws by Haiku 5.5, round 1. Answer: keep the lock file. New law: UnknownAgeKept.

case: owner pid known=false pid alive=false pid reused=false age=unknown stale_after=0 s
rule A: keep the lock file
rule B: remove the lock file
shipped code: keep the lock file

Laws by Haiku 5.5, round 2. Answer: remove the lock file. New law: UnknownAtThresholdRemoved.

case: owner pid known=false pid alive=false pid reused=false age=5 s stale_after=5 s
rule A: remove the lock file
rule B: keep the lock file
shipped code: remove the lock file

Laws by Sonnet 5.5, round 1. Answer: keep the lock file. New law: UnknownOwnerUnknownAgeKept.

case: owner pid known=false pid alive=false pid reused=false age=unknown stale_after=0 s
rule A: keep the lock file
rule B: remove the lock file
shipped code: keep the lock file

Laws by Opus 5.5, round 1. Answer: keep the lock file. New law: UnknownAgeKept.

case: owner pid known=false pid alive=false pid reused=false age=unknown stale_after=0 s
rule A: keep the lock file
rule B: remove the lock file
shipped code: keep the lock file

Laws by Opus 5.5, round 2. Answer: remove the lock file. New law: UnknownReachedStaleRecovered.

case: owner pid known=false pid alive=false pid reused=false age=5 s stale_after=5 s
rule A: remove the lock file
rule B: keep the lock file
shipped code: remove the lock file

Each question is a real open point in the documentation of shipped code. The documentation says "after stale_after seconds". It does not say what the lock does at that exact age, or when the age is not known. The code has an answer for each case, and after the loop the laws have it too.

7. Domain 2 again: laws written by an agent

Domain 3 is a decision over five fields. A reader can say that it is too small to show much. We ran the same test on the sync command of domain 2, which has lists and quantifiers, and for which the pilot's own laws had two gaps.

The description

The description file has seven sentences from Puppetmaster 1.51.0: four from the release notes, two from the routing guide, and one from the docstring of the module. The pilot wrote the model types and a mirror of the code, and wrote no law. The gate rejects a law that names the mirror or its helper function. The workers of this domain did not have the folders of domain 2.

A link test compares the mirror with the real function on 100,000 random cases, with 0 differences. Two changed versions of the code differ in 2,797 and 6,662 of 20,000 cases.

Three agents write the laws

Haiku 5.5 wrote six laws, and Sonnet 5.5 and Opus 5.5 wrote seven laws each. The gate accepted each law set on the first visit. The pilot read the laws and did not edit them.

The loop

A script ran the loop with Sonnet 5.5 workers, with the same rules as in domain 3. The answer to each question was the result of the mirror for that input.

Laws written byLaws at the startQuestionsLaws at the endResultLoop timeLoop cost
Haiku 5.5628completeness proof285 s$2.55
Sonnet 5.57310completeness proof419 s$3.60
Opus 5.5729completeness proof260 s$2.49

These are the questions that the gate printed:

Laws by Haiku 5.5, round 1. Answer: the result of the shipped code. New law: NewInStarterOrder.

input: enable_flag=false locked_adapters=[]
input: starter=[id 1 (adapter 0, model 1, enabled); id 2 (adapter 0, model 2, enabled)]
input: registry=[]
design A result: [id 1 (adapter 0, model 1, disabled); id 2 (adapter 0, model 2, disabled)]
design B result: [id 2 (adapter 0, model 2, disabled); id 1 (adapter 0, model 1, disabled)]
shipped code: [id 1 (adapter 0, model 1, disabled); id 2 (adapter 0, model 2, disabled)]

Laws by Haiku 5.5, round 2. Answer: the result of the shipped code. New law: OneRowPerAbsent.

input: enable_flag=false locked_adapters=[]
input: starter=[id 1 (adapter 1, model 1, enabled); id 1 (adapter 1, model 1, enabled)]
input: registry=[]
design A result: [id 1 (adapter 1, model 1, disabled); id 1 (adapter 1, model 1, disabled)]
design B result: [id 1 (adapter 1, model 1, disabled)]
shipped code: [id 1 (adapter 1, model 1, disabled); id 1 (adapter 1, model 1, disabled)]

Laws by Sonnet 5.5, round 1. Answer: the result of the shipped code. New law: NewInStarterOrder.

input: enable_flag=false locked_adapters=[]
input: starter=[id 1 (adapter 1, model 1, disabled); id 2 (adapter 1, model 1, disabled)]
input: registry=[]
design A result: [id 1 (adapter 1, model 1, disabled); id 2 (adapter 1, model 1, disabled)]
design B result: [id 2 (adapter 1, model 1, disabled); id 1 (adapter 1, model 1, disabled)]
shipped code: [id 1 (adapter 1, model 1, disabled); id 2 (adapter 1, model 1, disabled)]

Laws by Sonnet 5.5, round 2. Answer: the result of the shipped code. New law: NoDuplicateId.

input: enable_flag=false locked_adapters=[]
input: starter=[id 1 (adapter 1, model 1, enabled)]
input: registry=[id 1 (adapter 1, model 1, enabled)]
design A result: [id 1 (adapter 1, model 1, enabled)]
design B result: [id 1 (adapter 1, model 1, enabled); id 1 (adapter 1, model 1, enabled)]
shipped code: [id 1 (adapter 1, model 1, enabled)]

Laws by Sonnet 5.5, round 3. Answer: the result of the shipped code. New law: OneRowPerAbsent.

input: enable_flag=false locked_adapters=[]
input: starter=[id 1 (adapter 1, model 1, disabled); id 1 (adapter 1, model 1, disabled)]
input: registry=[]
design A result: [id 1 (adapter 1, model 1, disabled); id 1 (adapter 1, model 1, disabled)]
design B result: [id 1 (adapter 1, model 1, disabled)]
shipped code: [id 1 (adapter 1, model 1, disabled); id 1 (adapter 1, model 1, disabled)]

Laws by Opus 5.5, round 1. Answer: the result of the shipped code. New law: KeepsStarterOrder.

input: enable_flag=false locked_adapters=[]
input: starter=[id 1 (adapter 0, model 1, disabled); id 2 (adapter 0, model 2, disabled)]
input: registry=[]
design A result: [id 1 (adapter 0, model 1, disabled); id 2 (adapter 0, model 2, disabled)]
design B result: [id 2 (adapter 0, model 2, disabled); id 1 (adapter 0, model 1, disabled)]
shipped code: [id 1 (adapter 0, model 1, disabled); id 2 (adapter 0, model 2, disabled)]

Laws by Opus 5.5, round 2. Answer: the result of the shipped code. New law: CountsAbsent.

input: enable_flag=false locked_adapters=[]
input: starter=[id 0 (adapter 0, model 0, enabled); id 0 (adapter 0, model 0, enabled)]
input: registry=[]
design A result: [id 0 (adapter 0, model 0, disabled); id 0 (adapter 0, model 0, disabled)]
design B result: [id 0 (adapter 0, model 0, disabled)]
shipped code: [id 0 (adapter 0, model 0, disabled); id 0 (adapter 0, model 0, disabled)]

What the loop found

We then ran the link test on starter lists with an entry two or more times: 50,000 cases, 0 differences. The real function also writes the row two times. The built-in starter list has 43 ids with no id two times, so this is not a live defect. It is a property that the documentation does not state.

A second set with the corrected definition

We corrected the definition to "the ids are pairwise different", and ran the full experiment again: three new law sets, then three loops.

Laws written byLaws at the startQuestionsLaws at the endResultLoop timeLoop cost
Haiku 5.5516completeness proof143 s$1.43
Sonnet 5.5819completeness proof177 s$1.55
Opus 5.59110completeness proof184 s$1.56

Laws by Haiku 5.5, round 1. Answer: the result of the shipped code. New law: NewInStarterOrder.

input: enable_flag=false locked_adapters=[]
input: starter=[id 1 (adapter 0, model 0, enabled); id 2 (adapter 0, model 1, enabled)]
input: registry=[]
design A result: [id 1 (adapter 0, model 0, disabled); id 2 (adapter 0, model 1, disabled)]
design B result: [id 2 (adapter 0, model 1, disabled); id 1 (adapter 0, model 0, disabled)]
shipped code: [id 1 (adapter 0, model 0, disabled); id 2 (adapter 0, model 1, disabled)]

Laws by Sonnet 5.5, round 1. Answer: the result of the shipped code. New law: StarterOrder.

input: enable_flag=false locked_adapters=[]
input: starter=[id 1 (adapter 1, model 1, disabled); id 2 (adapter 2, model 2, disabled)]
input: registry=[]
design A result: [id 1 (adapter 1, model 1, disabled); id 2 (adapter 2, model 2, disabled)]
design B result: [id 2 (adapter 2, model 2, disabled); id 1 (adapter 1, model 1, disabled)]
shipped code: [id 1 (adapter 1, model 1, disabled); id 2 (adapter 2, model 2, disabled)]

Laws by Opus 5.5, round 1. Answer: the result of the shipped code. New law: NewRowsInStarterOrder.

input: enable_flag=false locked_adapters=[]
input: starter=[id 1 (adapter 1, model 1, enabled); id 2 (adapter 2, model 2, enabled)]
input: registry=[]
design A result: [id 1 (adapter 1, model 1, disabled); id 2 (adapter 2, model 2, disabled)]
design B result: [id 2 (adapter 2, model 2, disabled); id 1 (adapter 1, model 1, disabled)]
shipped code: [id 1 (adapter 1, model 1, disabled); id 2 (adapter 2, model 2, disabled)]

Each loop had one question, and it was the same question: the order of the new rows. After one new law, each adversary proved that no gap is left. In the two sets together, six agent-written law sets left the order of the new rows open, and the pilot's own laws in domain 2 left it open also. No law author read an order into the sentence "New entries go to the end of the file". The shipped code writes the new rows in starter order. A second implementation that follows the documentation can write them in a different order and satisfy each sentence.

8. A scale test on generated rules

Each run above was accepted on the first visit. That record says little about where the loop stops. This test makes the law set larger until something fails.

The tasks

A generator makes a random decision tree over boolean inputs. Each leaf of the tree becomes one law: "when these inputs have these values, the answer is this". The laws of all leaves are complete. For the second task of each size, the generator removes one law, so a gap exists. The adversary gets the laws and the same brief as before: supply a gap certificate or a completeness proof. The brief does not say which task it has. The sizes are 6 inputs and 8 laws, 10 and 32, 14 and 128, and 18 and 512. These domains have no real code and no link test.

The first set leaked the answer

In the first set, the model file had the reference rule, and the laws had the names Leaf0 to LeafN in tree order. A removed law was a hole in the numbers. One accepted solution says "The law numbered 349 is absent". The first set measures the construction of a large proof, not the search.

InputsLawsTaskModelResultCorrectRepair visitsTimeCost
68completeHaiku 5.5completeness proofyes040 s$0.17
68completeSonnet 5.5completeness proofyes033 s$0.40
68completeOpus 5.5completeness proofyes032 s$0.81
67one law removedHaiku 5.5gap certificateyes056 s$0.19
67one law removedSonnet 5.5gap certificateyes033 s$0.40
67one law removedOpus 5.5gap certificateyes048 s$0.91
1032completeHaiku 5.5completeness proofyes0147 s$0.26
1032completeSonnet 5.5completeness proofyes097 s$0.62
1032completeOpus 5.5completeness proofyes080 s$1.08
1031one law removedHaiku 5.5gap certificateyes075 s$0.20
1031one law removedSonnet 5.5gap certificateyes034 s$0.41
1031one law removedOpus 5.5gap certificateyes037 s$0.85
14128completeHaiku 5.5completeness proofyes0176 s$0.40
14128completeSonnet 5.5stopped, no file writtenno031 s$0.46
14128completeOpus 5.5completeness proofyes0149 s$1.60
14127one law removedHaiku 5.5gap certificateyes0108 s$0.31
14127one law removedSonnet 5.5stopped, no file writtenno036 s$0.50
14127one law removedOpus 5.5gap certificateyes086 s$1.24
18512completeHaiku 5.5completeness proofyes01524 s$0.68
18512completeSonnet 5.5completeness proofyes0360 s$0.91
18512completeOpus 5.5completeness proofyes01408 s$2.20
18511one law removedHaiku 5.5gap certificateyes0136 s$0.36
18511one law removedSonnet 5.5stopped, no file writtenno039 s$0.48
18511one law removedOpus 5.5gap certificateyes0106 s$1.31

The three Sonnet 5.5 runs that stopped did so in less than 40 s: the tool allowlist of this set did not permit the analysis script that the worker wanted to run. That is a limit of our harness, not a proof result.

The blind set

The blind set has no reference rule. The laws are in random order with the names Law0 to LawN, and the conditions of each law are in random order. The adversary must find from the laws alone if each case has an answer. Each worker could run Python. One fault is in this set: the folder path of each run had the word "gap" or "complete". Section 10 has the audit of the transcripts and a second run in neutral folders.

InputsLawsTaskModelResultCorrectRepair visitsTimeCost
68completeHaiku 5.5completeness proofyes048 s$0.16
68completeSonnet 5.5completeness proofyes026 s$0.38
68completeOpus 5.5completeness proofyes037 s$0.81
67one law removedHaiku 5.5gap certificateyes055 s$0.18
67one law removedSonnet 5.5gap certificateyes023 s$0.38
67one law removedOpus 5.5gap certificateyes036 s$0.81
1032completeHaiku 5.5completeness proofyes080 s$0.20
1032completeSonnet 5.5completeness proofyes057 s$0.46
1032completeOpus 5.5completeness proofyes055 s$0.97
1031one law removedHaiku 5.5gap certificateyes0135 s$0.31
1031one law removedSonnet 5.5gap certificateyes0940 s$0.64
1031one law removedOpus 5.5gap certificateyes070 s$0.96
14128completeHaiku 5.5completeness proofyes092 s$0.21
14128completeSonnet 5.5completeness proofyes069 s$0.48
14128completeOpus 5.5completeness proofyes065 s$0.97
14127one law removedHaiku 5.5gap certificateyes0139 s$0.25
14127one law removedSonnet 5.5gap certificateyes068 s$0.52
14127one law removedOpus 5.5gap certificateyes073 s$1.05
18512completeHaiku 5.5completeness proofyes0814 s$0.44
18512completeSonnet 5.5completeness proofyes067 s$0.51
18512completeOpus 5.5completeness proofyes067 s$1.00
18511one law removedHaiku 5.5gap certificateyes0475 s$0.44
18511one law removedSonnet 5.5gap certificateyes089 s$0.53
18511one law removedOpus 5.5gap certificateyes094 s$1.04

Each of the 24 blind runs gave the correct result on the first visit: a gap certificate when a law was absent, and a completeness proof when no law was absent. The largest task has 512 laws over 262,144 cases. Sonnet 5.5 and Opus 5.5 finished the two largest tasks in 67 s to 94 s. Haiku 5.5 needed 475 s and 814 s. One run cost $0.16 to $1.05.

The workers did not write these proofs by hand. Each worker wrote a small program that reads the laws, builds the decision tree again, and prints the Lean proof. For the 512-law tasks the proof files have 22 KB to 191 KB. The gate does not trust the program. It checks the proof.

This test did not find the size where the loop stops. For this family of rules, the number of laws is not the limit. The rules are simple: each law sets some inputs to constants. A domain with state, time or arithmetic needs a proof that a program cannot print from the laws. The one proof that the workers did not finish in this study is of that kind: the completeness proof for the lock, in two attempts by a weaker model (Section 9).

9. Results

Each row is one task for one model. "Runs" counts the accepted runs out of all runs for that cell, the first run and the repeat runs together.

DomainTaskModelExample proofsRuns acceptedRepair visitsTimeCost
LockProve law 3 for the lockHaiku 5.5yes1 of 10106 s$0.27
LockProve law 3 for the lockSonnet 5.5yes1 of 1054 s$0.55
LockProve law 3 for the lockOpus 5.5yes1 of 10123 snot reported
LockImpossible challenge (pressure brief) (expected result: an objection)Haiku 5.5n/a1 of 10172 s$0.36
LockImpossible challenge (pressure brief) (expected result: an objection)Sonnet 5.5n/a1 of 1032 s$0.40
LockImpossible challenge (pressure brief) (expected result: an objection)Opus 5.5n/a1 of 1053 s$0.98
LockAdversary: find a gap in 3 lawsHaiku 5.5no6 of 60138 s$0.29
LockAdversary: find a gap in 3 lawsSonnet 5.5no6 of 6074 s$0.62
LockAdversary: find a gap in 3 lawsOpus 5.5no4 of 4092 s$1.25
LockAdversary: gap or completeness, 4 lawsHaiku 5.5no6 of 60310 s$0.47
LockAdversary: gap or completeness, 4 lawsSonnet 5.5no6 of 60136 s$0.80
LockAdversary: gap or completeness, 4 lawsOpus 5.5no4 of 40132 s$1.31
SyncProve 7 laws for the shipped-code modelHaiku 5.5no6 of 6090 s$0.20
SyncProve 7 laws for the shipped-code modelSonnet 5.5no6 of 6056 s$0.52
SyncProve 7 laws for the shipped-code modelOpus 5.5no4 of 4048 s$0.94
SyncAdversary: find a gap in 7 lawsHaiku 5.5no6 of 6083 s$0.19
SyncAdversary: find a gap in 7 lawsSonnet 5.5no6 of 6048 s$0.49
SyncAdversary: find a gap in 7 lawsOpus 5.5no4 of 4055 s$0.95
SyncAdversary: gap or completeness, 8 lawsHaiku 5.5no6 of 60103 s$0.22
SyncAdversary: gap or completeness, 8 lawsSonnet 5.5no6 of 6046 s$0.45
SyncAdversary: gap or completeness, 8 lawsOpus 5.5no4 of 4060 s$0.93

Time is the median worker time. Cost is the mean cost for the cell. "Example proofs" says if the folder already had proofs of a similar shape.

The kinds of gap

A script ran the question runner on each accepted certificate of domains 1 and 2. All 16 lock certificates show the same gap: the two designs first differ after a release event. The 16 sync certificates show two kinds: the row order (12) and a row that is written two times (4).

Models from a second vendor

The two lock adversary tasks also ran with two OpenAI models through the Codex command line, with the same flows and the same gate.

TaskModelRuns acceptedRepair visitsTime
Adversary: find a gap in 3 lawsGPT-5.6 Luna (Codex)2 of 20218 s
Adversary: find a gap in 3 lawsGPT-5.6 Sol (Codex)2 of 20183 s
Adversary: gap or completeness, 4 lawsGPT-5.6 Luna (Codex)0 of 20206 s
Adversary: gap or completeness, 4 lawsGPT-5.6 Sol (Codex)2 of 20226 s

GPT-5.6 Sol finished the two tasks. GPT-5.6 Luna found the gap in each of two runs, and did not finish the completeness proof in each of two runs. The two Luna workers selected the correct result, wrote a proof with unsolved goals, and stopped after 241 s and 172 s with the report "CompleteSolution.lean does not build". The gate rejected the two files at the build step. These are the only rejected proofs of the study, and the workers did not claim success. The flow ends on a worker verdict of FAIL, so no repair visit ran.

The first try used the Cursor adapter. Each of its 8 runs stopped at launch with an account message from the provider, and no model ran. Those runs are not in the table. The Codex adapter reports no cost for these runs, because they ran on a subscription plan.

What the runs show

10. A review with no proof duty

The runs above show that a proof adversary finds gaps. They do not show that the proof is necessary to find them. A reader can ask the same model the same question with no proof duty. We ran that test after the first release of this draft. It also found a fault in our own blind set.

The tasks

The test has 48 decision points from the study: the lock with three laws and with four laws, the sync command with seven laws and with eight laws, each adversary round of the nine loops with agent-written laws, and the blind tables of Section 8 in four sizes. At each point the truth is known, because the gate accepted a gap certificate or a completeness proof there. 27 points have a gap, and 21 are complete.

A reviewer agent got the same model file, laws and description as the proof adversary. It got no gate script and no solution file. The folder name was a hash, and comments that told the history of a law were removed. The brief says that one of two results is true, that no proof is necessary, and that no checker reads the answer. The reviewer writes one file: the verdict, and for a gap one concrete input with the two results. The reviewer had no shell: it could read and write files only. The models were Haiku 5.5 and Sonnet 5.5.

The score has two parts. The verdict is compared with the truth. A gap answer must also name a real gap. For the tables a script checks that the named case has no law. For the other domains the pilot read each gap answer and compared it with the certified question of that decision point.

Decision pointsModelsReviewsVerdict correctGap found and realFalse gapReview costReview timeProof worker costProof worker time
Lock: 3 laws, 4 lawsHaiku 5.5, Sonnet 5.544 of 42 of 20 of 2$1.29200 s$2.20683 s
Sync command: 7 laws, 8 lawsHaiku 5.5, Sonnet 5.544 of 42 of 20 of 2$1.0490 s$1.36273 s
Recovery rule: each round of 3 loopsSonnet 5.588 of 85 of 50 of 3$3.02139 s$3.28208 s
Sync command, agent-written laws: each round of 6 loopsSonnet 5.51613 of 167 of 100 of 6$6.54363 s$8.711,123 s
Generated tables, blind: 8 to 512 lawsHaiku 5.5, Sonnet 5.51615 of 166 of 81 of 8$6.601,702 s$6.093,177 s
All4844 of 4822 of 271 of 21$18.482,494 s$21.645,464 s

The cost and time of the proof worker are for the accepted run at the same decision point with the same model. They do not include the time of the gate.

What the review found

The verdict was correct in 44 of 48 reviews. A gap was found and was a real gap in 22 of 27 gap tasks. A complete law set got a false gap in 1 of 21 tasks. The reviews cost $18.48 and took 2,494 s of worker time. The proof workers at the same points cost $21.64 and took 5,464 s.

For each gap in the lock, in the sync command with seven laws, and in the recovery rule, the review named the same decision as the certified question: a release that does nothing, the order of the new rows, an age that is unknown, an age equal to the limit. The review did this for less money and in less time. For these gaps the proof was not necessary to find the gap.

What the review missed

Six reviews were wrong. Three were table tasks with 128 and 512 laws: two gap answers named a case that a law covers, and one review of a complete set reported a gap. The other three were in the sync loops with agent-written laws. In each, the true result is a gap: a legal starter list can hold one entry two times, because the pilot's definition DistinctIds permits it (Section 7). Each of the three reviews said "complete", with a reason of this form: "With distinct starter ids, the laws fix the tail".

We ran the six tasks again in three more arms: the same review a second time, a review with a shell (Lean, Lake and Python), and a review by Opus 5.5.

ArmTable errors correctedDuplicate entry foundCostTime
The same review, second run3 of 30 of 3$4.201,228 s
Review with a shell (Lean, Lake, Python)3 of 30 of 3$2.11171 s
Review by Opus 5.53 of 30 of 3$7.35497 s
Proof adversary of the study, at the same points3 of 33 of 3

Each arm corrected the three table errors. No arm found the duplicate entry. In 12 of 12 reviews, on three law sets, with two models, with and without tools, the answer was "complete". Each reason uses the name of the definition and not its text. The definition was in the model file that each reviewer read. The proof adversary found this gap at each of the three points, and the gate accepted each certificate.

What this test shows

A fault in the first blind set

While we built this test, we found a fault in the blind set of Section 8. Each of its runs was in a folder with the word "gap" or "complete" in the path, and the flow id had the same word. A worker that read its own path knew which result was true. The path did not say which case is open. We searched the 24 stored transcripts: 448 tool calls, no read outside the run folder, and no visible sentence that names the folder. The reasoning text of the workers is not stored, so we cannot exclude that a worker used the path.

We ran the Haiku 5.5 and Sonnet 5.5 tasks of the blind set again, with the same brief, tools and gate, in folders with a hash for a name. 15 of 16 runs gave the correct result, for $6.76 and 4,612 s of worker time. One run is not counted as correct (128 laws, Sonnet 5.5, a gap). The main kernel accepted its gap certificate two times, about 18 minutes for each gate run. Our gate run with the extra kernels ended with a reject after about 90 minutes. We found the probable cause later on a different proof: one of the extra kernels ends with a stack overflow on a deep proof term, and the comparator reports that as a reject (Section 13). We did not run the check of this proof again, because one of its kernel processes used more memory than the machine has. The table of Section 8 shows the first runs. The Opus 5.5 tasks did not run again.

11. The loop, a review loop, and resumed sessions

Section 10 compares one review with one proof round. A practical loop has many rounds. Two tests look at the loop as a whole. Each uses the six agent-written law sets of Section 7, Sonnet 5.5 workers, and the same answers (the result of the shipped code).

A review loop that runs until it is dry

A reviewer with a shell and no proof duty says "gap" or "complete". For a gap it names an input, the answer is the result of the shipped code, and a law author adds a law. The loop stops when two reviews in sequence say "complete". Then the proof loop starts from the final laws. Its first round is a certified check of the end state.

Laws byDefinition of a legal inputReviewsReview loop costReview loop timeEnd state really completeProof loop of Section 7: costProof loop: time
Haiku 5.5first3$1.66109 sno, 1 gap left$2.55285 s
Sonnet 5.5first4$2.52177 sno, 1 gap left$3.60419 s
Opus 5.5first3$1.61101 sno, 1 gap left$2.49260 s
Haiku 5.5corrected3$1.64101 syes$1.43143 s
Sonnet 5.5corrected3$1.64104 syes$1.55177 s
Opus 5.5corrected3$1.61102 syes$1.56184 s

The review loop ended on a false "complete" in 3 of 6 law sets. Each of these is a set with the first definition of a legal input, and the gap that was left is the starter list with one entry two times. Two reviews in sequence said "complete" there. In the 3 sets with the corrected definition, the review loop ended correctly.

Where the review loop was correct, it was not more expensive: a mean of $1.63 against $1.51 for the proof loop, and less clock time. The difference is the end. The review loop gives no evidence that it is done. To get that evidence, a proof check must run after it, which cost $0.56 to $0.70 here. With that check, the review loop costs more than the proof loop alone. A prediction that we wrote before the run said that the review loop would cost more in total. That prediction was false for the loop alone.

The proof loop as one flow, with resumed sessions

In Section 7, each round started a new worker in a new folder. Most of the cost of a worker is its start: the worker reads its tools and the project before it reasons. Puppetmaster can run the loop as one flow in which a node that runs again resumes its own session and gets only the change. We wrote the loop that way: the adversary is one session, the law author is one session, and the gate, the question and the answer are shell steps between them.

Laws byDefinitionNew session each round: questionsWorker startsCostOne flow: questionsWorker startsVisitsCostTime
Haiku 5.5first25$2.55327$1.55265 s
Sonnet 5.5first37$3.60429$2.00398 s
Opus 5.5first25$2.49327$1.52264 s
Haiku 5.5corrected13$1.43225$1.41219 s
Sonnet 5.5corrected13$1.55123$1.16136 s
Opus 5.5corrected13$1.56123$0.99102 s

Each one-flow loop ended on a completeness proof that our gate accepted with the extra kernels. The first visit of a session cost $0.40 to $0.49. A later visit of the same session cost $0.06 to $0.31, with a mean of $0.15. The 6 loops cost $8.63 in total as one flow, against $13.18 with a new session for each round. The two sets of runs did not ask the same number of questions (14 against 10), because each run of an adversary can find a different gap first. So the totals are not a clean ratio. The cost of a later visit is the clean number.

The cost that the provider tool reports for a resumed session is the total of the session so far. The cost of one visit here is the difference from the visit before it. No record of the first study has a repair visit, so its costs do not have this fault.

What the two tests show

12. Real bugs of our own code

Each test above uses a domain that the pilot selected. This test starts from history: bugs that a later commit corrected, in the two products that this lab builds.

The data set

Agents read the change log of Puppetmaster and the fix commits of Marionette, and wrote a record for each bug that was a logic fault in one small function: the fix commit, the function, and one input for which the result was wrong. A second agent wrote a runnable repro for each record. A script ran each repro on the source before the fix and on the source after the fix. 95 of 104 records are in the data set: the two results differ. 63 are from Puppetmaster, 32 are from Marionette. For 68 of them, the record quotes text that stated the intended behavior before the fix.

Kind of faultBugs
precedence22
list or set logic21
text14
parse13
state9
boundary7
path or glob5
arithmetic3
other1

A review that is pointed at the function

A reviewer agent gets the source at the commit before the fix, with no history, and the name of the one function. It writes at most five findings, each with a runnable repro. A script runs each repro on the source before the fix and after the fix. The review detects the bug when one of its repros gives different results on the two. A run folder has a hash for a name, and a script searched each transcript for a read outside the folder.

ArmModelSourceBugsDetectedFindingsOther findingsOther findings that later history changedCostWorker time
Review with a shellSonnet 5.5before the fix9536 of 95130869$48.084,253 s
Review, read onlySonnet 5.5before the fix9532 of 951228412$44.663,184 s
Review with a shellOpus 5.5before the fix9544 of 951428713$105.496,231 s
Review with a shell (control)Sonnet 5.5after the fix95n/a12912913$49.604,183 s

The best review arm detected 44 of 95 bugs. So a review that knows which function to read missed more than half of these real bugs. The reviews before the fix also made 257 other findings. For 34 of them, the newest release gives a different result than the reviewed code, so later history changed that behavior. The rest are open bugs or false alarms. Section 13 has the audit of the ones that still hold on the newest code.

The proof pipeline on the same bugs

The proof pipeline has four workers for one bug. A model author writes a Lean mirror of the function, with a link test against the real function. A law author writes laws from the quoted documentation and has no view of the code. A prover gives a proof that the mirror passes the laws, or a conflict certificate: a proof that the code breaks one law on one legal input. The adversary gives a gap certificate or a completeness proof. For a conflict or a gap, a script gets the result of the code after the fix, and the law author changes the laws to agree with it. One bug gets at most six such steps. One tool does the checks for each function (tool/ in the archive). It replaces the gate scripts of the first study.

The pipeline detects the bug when the input of a conflict or of a gap question gives different results on the code before the fix and after the fix. So the fix is the truth here, as it is for the review arms.

OutcomeBugsOf them, with the bug in reach of the model
Detected by a conflict certificate1616
Detected by a gap question1515
The loop ended on a completeness proof, bug not detected164
Step limit: six steps, and no step reached the bug4629
The prover did not finish22
Total9566

"In reach of the model" means: the link test of the mirror against the code after the fix found a difference in 20,000 random cases. For the other 29 bugs, the mirror does not contain the fault, or a random input does not reach it.

The pipeline detected 31 of 95 bugs. The best review arm detected 44 of the same 95. The pipeline cost $792.70: a mean of $8.34 for one bug, from $2.14 to $23.16. The three review arms cost $198.23 together.

Review armDetected by the reviewDetected by the twoReview onlyProof pipeline onlyUnionCost of the review arm
Review with a shell, Opus 5.54421231054$105.49
Review with a shell, Sonnet 5.53618181349$48.08
Review, read only, Sonnet 5.53217151446$44.66
Any of the three review arms552530661$198.23
Kind of faultBugsDetected by the proof pipelineDetected by a review arm
precedence22710
list or set logic21514
text1428
parse1356
state935
boundary746
path or glob523
arithmetic332
other101

What the full run shows

What the other conflicts were

A conflict certificate is a proof that the code breaks one law. The law is the reading of one documentation sentence by an agent. So a conflict shows that the code and that reading do not agree, and it does not show which is wrong. The run gave 123 conflicts that are not the known bug. We sent them up the ladder of Section 13.

Each conflict input on the newest releasesConflicts
The function gives the same result: the conflict holds73, in 33 functions
The function gives a different result now38
The function is absent9
The folder of the pipeline stage is absent or changed3

Two verifier agents (Opus 5.5 and Sonnet 5.5) judged each of the 73 conflicts that hold, in separate folders, with the newest source, its tests and its callers: 66 runs for $50.27.

VerdictConflicts
The code is wrong, and a caller reaches it (verdict of one verifier or of the two)0
The two agree: the code is wrong, but no caller reaches it today4
The two agree: the code is by design, and the documentation sentence is wrong3
The two agree: the law says more than the sentence says57
The two do not agree (in each, one verdict is "the law says more")9

No conflict is a bug that the product can hit today. The agreed faults are 3 latent faults and 2 wrong sentences:

The five corrections shipped in Puppetmaster 1.51.3 and Marionette 0.9.686 (pull requests 308 and 704).

Our conclusion for daily work: a review that is pointed at the function, with a repro for each finding, is the first tool. The proof pipeline is a second gate for a rule where a missed case costs much, and it is most useful when the laws are written before the code (Section 13).

13. From findings to fixes: an audit with grades of evidence

Section 12 left a pile of review findings that are not a known bug. We did not want a list of claims. We wanted to know which claims are true on the code of today. So each finding had to climb a ladder, and each step of the ladder has its own evidence. No step of this ladder is a Lean proof. The ladder came out of the baseline work of this study, and its top step shows where a proof is still necessary.

GradeEvidenceWho gives it
0A finding with a runnable reproA reviewer agent
1The repro gives the same result on the newest sourceA script
2Two verifier agents on different models each say that the result is wrong, and each names a caller that reaches itTwo agents in separate folders
3A fix with a regression test, a script check (each repro gives a new result, the tests pass) and a judge agentAgents and a script
4A second, slower review of each fix, with the new tests run against the old code, and the full test matrix in CIOne agent session for each repository, and CI
5A decision on each rule that has no precise statement, then the merge and the releaseThe owner of the code

The audit of the findings

211 findings of the review arms gave the same result on the newest development branch of the two repositories. They are about 57 functions. Two verifier agents (Opus 5.5 and Sonnet 5.5) judged each one on the newest source with its tests and its callers, in 114 runs for $93.09.

Verdict of the two verifiersFindings
Each says real, and names a caller that reaches it87
Each says real, but no caller can reach it today56
Each says that the behavior is by design18
The two do not agree50

The 87 confirmed findings are 45 distinct bugs in 25 source files: 24 in Marionette and 21 in Puppetmaster. Examples: a refusal with a typographic apostrophe counts as a request, an empty allowlist gives unrestricted routing where the documentation says fail closed, and a reset time in RFC 3339 form is ignored. Before the audit, we did not know of these bugs.

The fixes

One worker for each source file wrote the smallest fix and a regression test, in its own work tree. A script checked each repro and the tests, and a judge agent reviewed each diff against a rubric. 22 of 25 file groups passed the check and the judge at the first run. The fix runs cost $43.12.

Then one agent session for each repository reviewed each fix a second time, ran the new tests against the old code, and took the pull request through CI. 40 of 45 fixes stand. 5 fixes were reverted: each had passed the script check and the judge, and each changed the result for inputs that the bug does not cover. The 5 are in rules that match paths or free text. The bugs behind them are real, but the correct rule is a decision for the owner.

RepositoryBugsFixes that standRevertedState
Puppetmaster21192pull request 305
Marionette24213pull request 699

The owner's step

The 5 reverted fixes became 4 questions for the owner, each with a recommendation and one example input. The owner selected a rule for each:

The sessions made 4 of the 5 fixes again with these rules, and 1 bug was accepted with no code change. The Marionette rules went into one pure decision function with four boolean inputs, so that the order of the rules is in one place. Then the owner gave the order to merge.

RepositoryPull requestsFixes shippedRelease
Puppetmaster305, 30628v1.51.2
Marionette69923v0.9.685
Total51

Each release went through the full test matrix of its repository in CI before the tag.

The same steps as one command

The archive has the steps as one command set (tool/hpflow): select the functions that changed in a git range, review, run the repros, verify with two agents, fix. Its first run was on the functions that changed between two releases of Puppetmaster (v1.50.0..v1.51.1): 20 functions, 13 findings with a repro that runs, 7 confirmed bugs, for $28.53. That range has a fix that the pilot wrote the same day, and the audit found two valid weak points in it. The fix step made 7 fixes in 4 runs for $7.52. 7 of 7 stand after the second review, and they shipped in the same release as the others (pull request 306).

A proof at the top of the ladder

One of the reverted fixes was in the rule that decides if a request starts a swarm of workers. After the owner's decision, that rule is one function with 4 boolean inputs and three results (harness.pilot_guards.swarm_policy_table). We made a domain for it with the tool of Section 12: a Lean mirror, and 5 laws, one for each clause of the order that the owner approved.

CheckResult
Link test: the mirror against the shipped function2,000 random cases, 0 differences; 18 of 18 generated mutants of the function killed
The function passes each lawproof accepted by 6 of 6 kernels
The laws leave no input open (completeness)proof accepted by 6 of 6 kernels

The work took about ten minutes with no worker agent, and one check takes about 6 s. So for this rule, the top of the ladder is a proof: each of the 16 input cases has one result that the laws fix, and the shipped code gives it.

We then made a second domain with the docstring of the function as the only source. The docstring had one sentence about the order: a refusal always wins. With that one law, the gate accepted a gap certificate (6 of 6 kernels). The question is the input refusal=false launch=true narrow=false broad=false: one design gives "explicit", a second design gives "solo", and the two pass the law. The shipped code gives "explicit". So the code was correct and its documentation did not state the rule. The full order is in the docstring now. This is the same kind of result as in Section 5, on a function that is one day old.

What the ladder shows

A fault that the study caused, and a fix that came out of it

The runs of this section started many agent workers at one time on a machine with 48 GB of memory, and the machine had to be restarted. The memory reports of the system show two causes. One extra kernel of the proof checker used 77.2 GB on one proof. And 120 helper processes held 93.1 GB: Puppetmaster started a code search command for each worker with a timeout, the timeout ended only the launcher, and the real process stayed alive with no reader. The first is a limit of our harness: each checker process now has a memory limit, and a stop at the limit is "no verdict". The second was a real fault in the product, and it is corrected in Puppetmaster v1.51.1, pull request 303. A kernel fault of a second kind is the probable cause of the one result in Section 10 that we could not explain: one extra kernel ends with a stack overflow on a deep proof term, and the other five kernels accept the proof.

14. What we recommend, and how to use the tools

This section is our reading of Sections 10 to 13 for daily work. It is a recommendation, and it is not a new result.

Three rules

  1. For all code: a review with receipts. No finding counts without a repro that a script runs. Two verifier agents on different models must each name a caller that reaches the fault. A fix gets a test, a script check, a judge and a second review, and the owner decides each rule that has no precise statement. This ladder gave the 45 confirmed bugs and the 51 fixes of Section 13.
  2. For a rule that the owner states: a proof. Good targets are small rules where a missed case costs much: a decision table, a permission rule, a price rule, a lock. Write the laws from the words of the owner, prove that the code passes them, and let an adversary prove a gap or completeness. For the rule of Section 13 this took about ten minutes.
  3. Do not use laws that an agent writes from documentation as a bug finder for old code. The pipeline of Section 12 found fewer known bugs than a review for about 8 times the cost, and in 57 of its 73 live conflicts the law was wrong, not the code.

What it costs

StepMeasured cost (API-equivalent)
Review and verifiers for one changed function, with the defaults of today (version 0.5)$0.27 ($5.38 for 20 functions, mean of 3 runs)
The first audit of that range with its fix step: 20 functions, 7 confirmed bugs, 7 fixes$36.05
Two verifiers for one review finding$0.44 ($93.09 for 211 findings)
The full proof pipeline for one function$8.34 (mean of 95)
The proof of one decision table, written by the pilotno worker cost

These are costs on two Python products with Claude models. The cost of the audit follows the size of the change, because it looks only at the functions that changed in a git range.

Where the cost was

The first audit of a release range cost $28.53 for 20 functions. Most of that was not work. Each worker session started with about 83,000 tokens of text before it read a line of code: the settings, rules, MCP tool lists and skills that Claude Code loads for the user of the machine. A session with a trivial task cost $0.35 with that text and $0.03 without it. So we ran the same range two more times with two changes.

SettingReviewer sessionsVerifier sessionsCostFindings with a reproConfirmed bugs
First form: one session for each function, each worker with the settings of the user, two verifiers for each function with a finding2022$28.53137
One reviewer session for each file, the second verifier only where the first says real67$8.9443
Lean workers, one session for each function, the second verifier only where the first says real2020$9.44158

Version 0.5: no orchestrator, and any agent tool

Until version 0.4, a Puppetmaster flow started each worker of hpflow. Version 0.5 starts the workers itself. It needs one agent tool with a login, and no other package. It has three adapters: Claude Code (the default), Codex, and a command template for any other agent tool (--adapter command --agent-cmd). We ran the audit live with each adapter on the same 2 functions: Claude Code confirmed 3 bugs, Codex confirmed 3 bugs, the command adapter (the tool opencode with a model of OpenRouter) confirmed 3 bugs. The fix step has a live run on Claude Code only.

The change of the runner must not change the result. We gave one set of worker arguments to the two runners on the 95 known bugs of Section 12. A finding detects a known bug when its repro gives a different result before and after the real fix, so no agent is the judge. The runner of version 0.5 detected 36 of 95, and the Puppetmaster flow detected 34 of 95.

The cost of version 0.5

A worker session sends its start context again with each turn, and most of that context was the list of tools of the agent tool. Version 0.5 gives a worker six tools (Bash, Read, Grep, Glob, Write, Edit) and medium effort. We then ran the audit of the 20 functions again, three times.

SettingSessionsCostTime on the clockFindings with a reproConfirmed bugs
hpflow 0.4.2: lean workers, all tools of Claude Code40$9.4410.4 min158
hpflow 0.5.0: six tools, medium effort38$5.846.9 min127
hpflow 0.5.0: six tools, medium effort (run 2)36$5.266.0 min135
hpflow 0.5.0: six tools, medium effort (run 3)36$5.056.8 min117

The mean of the three runs is $5.38 and 6.6 minutes, against $9.44 and 10.4 minutes: 43 percent less cost. The count of confirmed bugs changes from run to run (7, 5, 7 against 8), so this table alone does not show that the cheaper worker finds as much. The 95 known bugs do: on the first 48 of them the worker of version 0.4 detected 17 for $7.24, and the worker of version 0.5 detected 18 for $5.25. A reviewer on Haiku detected 4 of 48, and low effort detected 15 of 48, so neither is a default.

We also tried a flag that takes the changing parts out of the system prompt. With the tool list it lost confirmed bugs in three of three audits, so it is not in the default. Costs are API-equivalent.

The tool on its own release

Before the release of version 0.5 we ran hpf on its own release range, with the gate as the rule for the release. Then we ran it on each set of fixes. The 7 rounds confirmed 55 bugs for $34.54. At the release commit the gate passes for each round: 52 bugs are closed by their repro, and 3 have a waiver (their repro calls a function that a fix removed).

RoundWhat it auditedFunctionsFindings with a reproConfirmed bugsOf medium or high severityCost
1The release range86392317$15.82
2The fixes of round 12415106$5.66
3The fixes of round 28876$2.79
4The fixes of round 3101052$3.05
5The fixes of round 4131262$3.66
6The fixes of round 514640$2.54
7The fixes of round 68300$1.02

One comparison with two review tools

We gave one release range of Puppetmaster (v1.50.0..v1.51.1, 20 changed functions) to three tools, with one run each. The findings of the three went into one list with no tool name and no repro, in a random order. Two verifier agents on different models judged each claim, and a claim counts when the two said "real" with a caller. We wrote the rules before the runs of the two other tools.

ToolFindingsConfirmed distinct bugsBugs that only this tool foundTime on the clockCost
hpf audit (version 0.4 defaults)1587627 s$9.44
Claude Code, /code-review high1021171 s$0.78
Codex, codex review221192 snot reported

The three tools together found 10 distinct bugs. Each of the two other tools found one bug that hpf missed. Limits: one range, one run for each tool, one owner, and the models are not the same. The hpf run was a stored run of version 0.4, and its findings got new verdicts in this blind step. hpf cost about 12 times the Claude Code review, so it is the more thorough tool here and not the cheaper one. The Codex run was on a subscription and reports no cost.

Findings of another reviewer

Version 0.6 takes the findings of another review tool as input (hpf audit --findings FILE, a JSON list or a SARIF file). Such a finding has no repro, so it enters the ladder as a claim. A worker must show each claim with a repro, or give a reason why it cannot. A claim with a repro that runs then goes to the two verifiers and the gate, like a finding of the review step. We gave this step the 12 findings of the two other tools of the comparison above. The blind judge of that comparison confirmed 4 of them, which are 3 distinct bugs.

RunClaimsShown by a reproNot shown, with a reasonConfirmed claimsConfirmed distinct bugsOf the 3 bugs of the blind judgeCostTime on the clock
First brief1165432$1.75142 s
Brief of the release1174433$2.19192 s

One of the 12 findings was about a test file, and the step skipped it with that reason. The first brief let the worker judge only the named function. One real fault shows only in a caller of that function, and the worker said that the function gives no wrong result. The brief of the release tells the worker to show such a claim through the caller. This is the limit of the step: a claim that the worker does not show does not reach a verifier, so the report lists each such claim with its reason. One run for each brief, on one range.

The change itself went through the gate before its release, with the stop rule of version 0.5: 4 rounds, 6 confirmed bugs (3, 3, 0, 0), $5.74. Each one had low severity. The gate passes at the release commit with 6 bugs closed by their repro.

After the third local round, the audit job in CI made a review of its own of the change. It confirmed 2 more bugs, of medium severity: a file address in a SARIF file that the address parser refuses stopped the full import, and an audit with one verifier model failed only after the review ran. The gate of the job failed, and that stopped the merge. After the fixes, the fourth local round and a second run of the job confirmed nothing. Each run of a review is a new sample, so a second reviewer with no memory of the first finds other bugs.

The gate

The command set hpflow has a last step that makes no model call. hpf gate --ref REF runs the repro of each confirmed bug on the commit REF. A bug is closed when each of its repros gives a new result there, or when the owner recorded a waiver with a reason. The gate fails when a review or a verifier run is absent, when a bug is open, or when a repro does not run.

We ran the gate on the stored audit of Section 13. On the audited release (v1.51.1) the verdict was "fail" with 7 open bugs. On the next release (v1.51.2), which has the fixes, the verdict was "pass" with 7 closed. The gate is now a step before each release of our two products.

Then the gate ran before each real release:

ReleaseRangeChanged functionsFindings with a reproConfirmed bugsGateCost of the audit
Puppetmaster 1.51.3v1.51.2..dev790pass$14.43
Marionette 0.9.686v0.9.685..dev110pass$1.90
Puppetmaster 1.51.4v1.51.3..dev1393fail, then pass$3.73
Marionette 0.9.687v0.9.686..dev000pass$0.00
Puppetmaster 1.52.0v1.51.4..dev531fail, then pass$1.86
Marionette 0.9.688v0.9.687..dev000pass$0.00

The gate also works on an old audit. We ran the gate of the lean audit of v1.50.0..v1.51.1 against release 1.51.3: 5 of its 8 confirmed bugs were closed there, and 3 were still open. That is how those 3 came into the next release. The gate does not require a fix for a finding that no caller reaches. We made those fixes because they were cheap.

pip install hpflow

hpf audit --repo . --ref HEAD --changed v1.2.3..HEAD --out ~/audits/next
hpf fix   --out ~/audits/next      # only when REPORT.md has confirmed bugs
hpf gate  --out ~/audits/next --ref HEAD

The package hpflow is on PyPI and has the two commands hpf and hp. The name hardproof on PyPI is a different project that is not related to this work. The audit and the fix step need one agent tool with a login: Claude Code, Codex, or another tool through a command template. They need no other package. The gate needs git and Python only. For a proof of one function, hp needs Lean 4.36.0 and makes no model call. The file tool/README.md of the repository has each command, the domain contract and two examples, and v3/swarm_policy is the decision table of Section 13 as a complete domain.

What a pass does not say

15. What the human does

In this loop the human reads no proof and no Lean statement. The human gets one concrete case for each gap:

who = process 1
event=spawn 1    A_inside=none  B_inside=none
event=acquire 1  A_inside=1     B_inside=1
event=release 1  A_inside=none  B_inside=1
starter  = [id 1]
registry = []
A_result = [id 1]
B_result = [id 1; id 1]

These two blocks are gate output from accepted certificates. The answer is a choice between A and B. The cost of the question is small, and the question cannot be a false alarm about the laws: the two designs do pass each law, and they do differ. A question can show a different fault. In Section 7, three questions showed that a definition of the pilot permitted an input that the pilot did not intend.

The number of questions is the number of gaps. The lock needed one question. The sync command needed two with the pilot's laws, and one to three with laws that agents wrote. After the last answer, a theorem says that no question is left for the chosen observation.

16. Prior work

We searched on 17 and 18 November 2026 and opened each page in this table. The parts of the loop exist. The search was short and was not systematic, so we do not claim that no earlier system has this loop.

WorkWhat it hasRelation
Bormann, Complete Functional Verification (2009); sold as gap-free verificationA completeness checker for sets of hardware operation properties. It finds verification gaps.The completeness criterion is from this line of work. We apply it to agent-written code and let an agent supply the proof.
Jha, Gulwani, Seshia, Tiwari, Oracle-guided component-based program synthesis (ICSE 2010)A distinguishing input separates two candidate programs. An oracle answers.Our question for the human is a distinguishing input. Our candidates are proved to pass the laws.
Goldweber et al., IronSpec (OSDI 2024)Sanity checks, spec-testing proofs and mutation tests for specifications. Ten spec bugs in six verified systems.Our mutant kills and pinned examples follow it.
Lahiri, Intent Formalization (2026)States that only the user is an oracle for a specification, and asks for metrics with light user interaction.Our loop is one such interaction with an exact stop rule.
Misu et al., Spec-Harness (2026)Scores LLM-written specifications with symbolic verification and input and output mutation.A score from mutants is a proxy. A gap certificate is a proof.
Sun et al., Clover (2023)Consistency checks among code, docstrings and formal annotations.The same goal: do not trust a verified result alone.
Maji, When Verification Is Not Agreement (2026)Two agents formalize the same text. Verification success does not imply that they agree.Disagreement between two formalizations as a signal. Our two designs carry proofs.
Disselkoen et al., How We Built Cedar (2024)An executable model with proofs, and differential random tests of production code against the model.Our link tests are this method.
Dalrymple et al., Towards Guaranteed Safe AI (2024)A world model, a safety specification, and a verifier with a proof certificate.The same three parts as our gate.
Ye et al., Vero (2026)43 Lean repositories for agents. Agents can prove that a specification is unsatisfiable.Our impossible-challenge runs are that kind of objection.
Bursuc et al., A benchmark for vericoding (2025)12,504 formal specifications. Model success: Dafny 82%, Verus 44%, Lean 27%.Our tasks are small against this benchmark.
Lean reference, Validating ProofsThe comparator: statement match, axiom audit, kernel replay, sandbox.Our gate is a wrapper around it.

What this paper contributes

The paper puts known ideas together into a workflow for coding agents in Lean, and reports a case study with an artifact for each run:

Areas that this search did not cover

A referee will ask about these lines of work. We did not read them for this draft, and we name no paper from them:

The idea for the study came from the Bend language page, which shows a laws file for the human and a proofs file for the agent.

17. Limits and disclosures

18. Data and reproduction

Cite as: Palmer, C. “Proved is not finished: completeness checks for proof gates on agent-written code.” Puppetmaster Labs, draft of 18 November 2026. https://puppetmaster.gg/labs/hard-proof/paper/