Loading world
voxelsPUPPETMASTER LABS · FOUR CONTROLLED STUDIES · NOV 2026
When does a pilotneed a crew?
One agent alone wins a small job. Past a point, the work splits into parts that can run at the same time, and a crew pulls ahead. Then the question is which crew. We ran it on a 16-region world. Then we made the job 4x bigger. The gap got wider. Then we measured where the crew starts to win: 16 regions. Then we put the two crews head to head, blind. From 16 regions up, the judges picked Puppetmaster.
Same model. Same effort. Only the crew changed.
On a 16-region and a 64-region task, Puppetmaster costs less and makes better worlds than Codex’s own subagents, with the same model and effort.
- Follow-up cost
- 16 regions−71.6%64 regions−85.1%
- Full task cost
- 16 regions−35.3%64 regions−41.5%
- Blind scene votes
- 16 regions9–064 regions42–2
- Pairs won on cost
- 16 regions4/464 regions4/4
gpt-6.1-sol on high in every worker, both sides. Four full pairs per size. Rules locked by SHA-256 before run one. Four separate studies. The fourth judged the third’s crew worlds again with fresh votes. Votes never get pooled across studies.
THE JOB
Repair a world. Then change part of it.
- Cold build. Broken region generators, one voxel world: 16 regions on a 64 by 64 world, or 64 regions on an 8 by 8 grid in a 128 by 128 world. Each region needs smooth terrain, a landmark with a real silhouette, roads that join its neighbors and water that belongs to its biome. One public judge checks every region and the whole world. Budget: 1,800 s at 16 regions, 3,600 s at 64.
- Follow-up. The same session gets new specs for some regions: three at 16 regions, six at 64. Change those. Leave everything else alone.
01 / THE WORLDS
Two crews, one model.
16 regions Fly through the real worlds
Loading world
voxels64 regions The judges’ renders
Pair 4, after the follow-up. These are the exact renders the judges saw: same camera, same seed, same renderer for both.


02 / THE VERDICT
Six rules. Written first.
Each study locked the same six rules and their order by SHA-256 before rep 1. A later rule can’t cancel an earlier one. The quality gate goes first: lose on quality and nothing else counts.
Before the 64-region run we called a cold-cost loss at that size. Wrong. That call is in the frozen declaration.
03 / COST PER PAIR
The follow-up is where the gap lives.
On the follow-up, Puppetmaster only re-ran the builders of the changed regions, then the checks for their groups and the world. Codex resumed 11 to 14 sessions at 16 regions and 10 to 55 at 64. Costs are API-equivalent USD from the provider’s usage records. Each size has its own scale.
16 regions
Show every 16-region phase as a table
| Pair | Arm | Phase | USD | Wall s | Input tokens | Cached | Output tokens | Q |
|---|
64 regions
Show every 64-region phase as a table
| Pair | Arm | Phase | USD | Wall s | Input tokens | Cached | Output tokens | Resumed | Q |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Codex subagents | Cold | $14.71 | 1,178.7 | 51,786,750 | 48,592,768 | 346,322 | 0 | 75 |
| 1 | Codex subagents | Follow-up | $6.21 | 1,010.1 | 27,721,386 | 26,377,600 | 88,526 | 10 | 75 |
| 1 | Puppetmaster | Cold | $13.78 | 988.3 | 32,108,016 | 28,939,904 | 455,040 | 0 | 70 |
| 1 | Puppetmaster | Follow-up | $1.62 | 307.4 | 7,455,585 | 7,168,512 | 33,206 | 14 | 75 |
| 2 | Codex subagents | Cold | $15.44 | 1,153.5 | 53,492,507 | 50,044,416 | 353,524 | 0 | 70 |
| 2 | Codex subagents | Follow-up | $12.27 | 1,436.8 | 55,763,167 | 53,471,616 | 234,197 | 39 | 70 |
| 2 | Puppetmaster | Cold | $14.41 | 1,340.8 | 35,665,377 | 32,441,984 | 471,519 | 0 | 70 |
| 2 | Puppetmaster | Follow-up | $1.46 | 288.8 | 6,794,551 | 6,551,936 | 31,796 | 14 | 70 |
| 3 | Codex subagents | Cold | $15.42 | 1,048.1 | 51,721,291 | 48,142,208 | 344,694 | 0 | 70 |
| 3 | Codex subagents | Follow-up | $9.97 | 919.0 | 41,363,174 | 39,157,760 | 164,775 | 55 | 75 |
| 3 | Puppetmaster | Cold | $13.97 | 1,235.2 | 32,747,962 | 29,478,400 | 448,438 | 0 | 70 |
| 3 | Puppetmaster | Follow-up | $1.73 | 310.6 | 6,572,533 | 6,187,648 | 33,795 | 14 | 75 |
| 4 | Codex subagents | Cold | $17.83 | 1,330.6 | 63,013,041 | 59,127,040 | 414,850 | 0 | 75 |
| 4 | Codex subagents | Follow-up | $14.54 | 1,148.3 | 59,628,583 | 56,374,784 | 238,993 | 31 | 75 |
| 4 | Puppetmaster | Cold | $13.65 | 1,007.6 | 31,977,688 | 28,946,176 | 469,606 | 0 | 75 |
| 4 | Puppetmaster | Follow-up | $1.62 | 254.4 | 7,285,742 | 7,005,696 | 35,637 | 22 | 78.75 |
At 64 regions, pair 4 had a failed group check, so 8 more builders went back in. Follow-up wall time, just for the record: 1,161 s total for Puppetmaster against 4,514 s for Codex. Rule 6 only scores the cold build.
04 / BLIND JUDGES
Two judges. Both orders. No names.
Claude Opus 5.5 and Kimi K3.1 Pro saw the same renders of both worlds as A and B. Neither judge family built a world. Each judge saw every scene twice, once in each order. A vote counts only if it picks the same world both times. A scene counts only if both judges agree.
16 regions 24 scenes
Select a scene to read what each judge wrote.
- Kimi picked whichever world it saw first in 39 of 48 calls. The two-order rule turned that bias into ties. Kimi never gave a consistent pick for Codex.
- Claude gave a consistent Puppetmaster pick in 23 of 24 scenes. Kimi agreed in 9. Kimi set the count.
- The separate rubric grade scored Codex higher in pair 4 (78.75 against 75.00). Both arms passed the Q70 bar in every phase.
64 regions 120 scenes
At judge size a 128 by 128 world makes every region tiny. So each phase gets 15 scenes: 3 whole-world shots and 12 quadrant close-ups.
| Criterion, both judges agree | Puppetmaster | Codex | Tie |
|---|---|---|---|
| Composition | 38 | 2 | 80 |
| Geometric craft | 42 | 1 | 77 |
| Integration finish | 11 | 1 | 108 |
| Palette material | 0 | 0 | 120 |
| Readability | 24 | 4 | 92 |
One decided scene, side by side
Pair 4 follow-up, quadrant 2-1, seed 1709. Both judges, both orders: Puppetmaster.


Claude, Puppetmaster shown second
"World B shows more articulated landmarks, including an arch, a walled keep with corner towers, and a detailed white tower."
Kimi, Puppetmaster shown first
"World A has more distinct landmark silhouettes, richer biome transitions, and a clearer connected path network."
- Claude likes whatever it sees second (131 picks to 77). Kimi likes whatever it sees first (130 to 110). Asking both orders turns that bias into a tie.
- After that: Claude 55 Puppetmaster, 10 Codex, 55 ties. Kimi 75, 9, 36. They agreed on 44 decided scenes.
- The separate rubric grade had Codex ahead in pair 1 cold (75 vs 70) and Puppetmaster ahead in pair 4 follow-up (78.75 vs 75). Everything else tied. Both sides cleared Q70 in every phase.
05 / INSIDE THE WORK
Each piece has a boundary.
The 16-region world: sixteen regions, eight landmark types. Select a region to fly both cameras to it. Gold outlines mark the three regions that the follow-up revised.
SELECT A REGION
A contract, a landmark, a place in the world.
PUPPETMASTER COLD RUN · 16 REGIONS · PAIR 4
Where the cold wall time goes
Against pair 4 of an earlier run: same Puppetmaster 1.48.0, same graph, older lab with per-worker guard checks. This study removed those checks. The comparison does not isolate one cause.
06 / HOW WE GOT HERE
We took the crew tax apart.
A crew pays a fixed price per worker: start-up, a cold read of the task, and review. Each release below went after one part of that price. The replica numbers are cold runs on one machine with no lab wrapper. Do not mix them with the study numbers.
07 / WHERE THE LINE IS
The judgment, in your hands.
Puppetmaster and Marionette do not fan out because a prompt sounds big. A sizing gate reads the plan and the measured pace, then decides. This is that gate: the same decision function as puppetmaster/sizing.py, checked against it on 2,611 cases. Move the sliders.
We measured the line. Solo against both crews · 4, 16, 64 regions
One agent alone, Puppetmaster and Codex subagents built the same world at three sizes. Three runs per arm per size, gpt-6.1-sol on high everywhere. Two blind judges compared each solo world with each crew world, scene by scene, in both orders. The rule was locked before run one: a crew wins a size only with more scene wins, an exact binomial p under 0.05, and a majority in at least 4 of 6 comparisons.
Solo never failed. It passed every check at every size, and it was the fastest and cheapest arm. The judges still preferred a crew from 16 regions up. At 16 regions, Puppetmaster won 25 scenes to 0 against solo. Codex subagents won 13 to 7 and did not clear the rule.
Then the crews, head to head. Puppetmaster against Codex subagents · same 18 worlds
We took the 18 crew worlds above and asked the same two blind judges a new question: which crew built the better world? Same scenes, both orders, every vote fresh. We said in advance that this result goes on this page whichever way it went. The rule was locked before the first vote: a crew is preferred only with more scene wins over all three sizes, an exact two-sided binomial p under 0.05, and a majority in at least 6 of 9 comparisons.
The locked test, over all three sizes: Puppetmaster preferred. Puppetmaster 39 scenes, Codex subagents 20, 76 ties. p = 0.018. Six of nine runs favored Puppetmaster, two favored Codex subagents, one was even. From 16 regions up, Codex subagents won 3 of 90 scenes. At 4 regions, Codex subagents led 17 to 8. That did not pass the test, and we do not hide it: at 4 regions a crew has no quality case over solo either. A tie does not prove the crews are equal. Judge cash: $7.69 for 541 calls.
The gate, live.
Below 4 independent units
Stay solo. One session that holds the whole problem beats any crew. A crew pays its start-up price on every worker.
Measure, then decide
The pilot starts alone. After the first finished units, the gate projects solo time against a handoff. It hands off only when the crew finishes sooner.
Many substantial units
At 16 or more independent units that each take real time, the gate hands off as soon as a crew projects to finish sooner. With a calibrated pace per unit, it starts with a crew.
The measured line agrees with the gate’s default of 16 independent units, so the default stays. Solo pace per region was mostly fixed start-up work (76 s per region at 4 regions, 18 s at 16), so the gate still measures pace live and does not take a fixed value from one task.
08 / LIMITS
What we publish with the claim.
Held equal
gpt-6.1-sol on high in every worker, both sides. Inside each study: the same world, plan, cold request and follow-up request, and the same work budget (1,800 s at 16 regions, 3,600 s at 64). No routing, no effort lanes, no cheaper models.
Changed
Only the crew. Codex: the CLI’s built-in subagents. Puppetmaster: one flow graph with per-group builders, checks and review. puppetmaster-ai 1.48.0 at 16 regions. 1.49.0 at 64, with 8 column groups and one builder per region.
Scope
One world family, one lab, four pairs per size. At 64 regions, one quota stop: the two hit attempts reran once on the same pin, and the first tries stay out of the sums. The solo study: three runs per arm at 4, 16 and 64 regions with puppetmaster-ai 1.49.0. Sizes 5 to 15 not tested. Eight of its 64-region worlds come from the 64-region study and were judged there before. Every vote in the solo study is fresh. The crew head-to-head used the solo study’s 18 crew worlds, with fresh votes and no new builds.
Money
Builder costs are API-equivalent prices for the recorded tokens. The two blind judges cost $3.83 in real OpenRouter cash at 16 regions, $7.92 at 64, $15.35 for the solo study (1,080 calls), and $7.69 for the crew head-to-head (541 calls).
Read the paper 16 regions (JSON)64 regions (JSON)Solo line (JSON)Crews head to head (JSON)Puppetmaster ↗
