PUPPETMASTER LABS · FOUR CONTROLLED STUDIES · NOV 2026

When does a pilotneed a crew?

One agent alone wins a small job. Past a point, the work splits into parts that can run at the same time, and a crew pulls ahead. Then the question is which crew. We ran it on a 16-region world. Then we made the job 4x bigger. The gap got wider. Then we measured where the crew starts to win: 16 regions. Then we put the two crews head to head, blind. From 16 regions up, the judges picked Puppetmaster.

Same model. Same effort. Only the crew changed.

On a 16-region and a 64-region task, Puppetmaster costs less and makes better worlds than Codex’s own subagents, with the same model and effort.

Follow-up cost
16 regions−71.6%64 regions−85.1%
Full task cost
16 regions−35.3%64 regions−41.5%
Blind scene votes
16 regions9–064 regions42–2
Pairs won on cost
16 regions4/464 regions4/4

gpt-6.1-sol on high in every worker, both sides. Four full pairs per size. Rules locked by SHA-256 before run one. Four separate studies. The fourth judged the third’s crew worlds again with fresh votes. Votes never get pooled across studies.

THE JOB

Repair a world. Then change part of it.

  1. Cold build. Broken region generators, one voxel world: 16 regions on a 64 by 64 world, or 64 regions on an 8 by 8 grid in a 128 by 128 world. Each region needs smooth terrain, a landmark with a real silhouette, roads that join its neighbors and water that belongs to its biome. One public judge checks every region and the whole world. Budget: 1,800 s at 16 regions, 3,600 s at 64.
  2. Follow-up. The same session gets new specs for some regions: three at 16 regions, six at 64. Change those. Leave everything else alone.

01 / THE WORLDS

Two crews, one model.

16 regions Fly through the real worlds

Pair
Phase

Loading world

voxels

Loading world

voxels

64 regions The judges’ renders

Pair 4, after the follow-up. These are the exact renders the judges saw: same camera, same seed, same renderer for both.

The Codex subagents' 64-region world after the follow-up in pair 4, seen from the front at an angle
Codex subagents · pair 4 follow-up
Puppetmaster's 64-region world after the follow-up in pair 4, seen from the front at an angle
Puppetmaster · pair 4 follow-up

02 / THE VERDICT

Six rules. Written first.

Each study locked the same six rules and their order by SHA-256 before rep 1. A later rule can’t cancel an earlier one. The quality gate goes first: lose on quality and nothing else counts.

Rule16 regions64 regions
1 · Quality gatePassBoth sides qualified in 4 of 4 pairs. Codex never won a blind majority.PassBoth sides qualified in 4 of 4 pairs. Codex never won a blind majority.
2 · Follow-up cost primaryHoldsLower in 4 of 4. Median $0.738 against $2.600. U = 0, exact p = 0.014.HoldsLower in 4 of 4. Median $1.62 against $11.12. U = 0, exact p = 0.014.
3 · Full task costHoldsLower in 4 of 4. $19.68 against $30.45, 35.3% lower.HoldsLower in 4 of 4. $62.24 against $106.39, 41.5% lower.
4 · Cold costHoldsLower in 4 of 4. $16.70 against $20.59, 18.9% lower.HoldsLower in 4 of 4. $55.81 against $63.40, 12.0% lower.
5 · Blind qualityHoldsScene majority in 4 of 4 pairs. 9 scenes to 0, 15 ties. Exact one-sided p = 0.002.HoldsScene majority in 4 of 4 pairs. 42 scenes to 2, 76 ties. Exact one-sided p = 5.6 × 10⁻¹¹.
6 · Cold wall timeEvenFaster in 2 of 4. 1,907 s against 2,100 s in total. No difference by the rule.EvenFaster in 2 of 4. 4,572 s against 4,711 s in total. No difference by the rule.

Before the 64-region run we called a cold-cost loss at that size. Wrong. That call is in the frozen declaration.

03 / COST PER PAIR

The follow-up is where the gap lives.

On the follow-up, Puppetmaster only re-ran the builders of the changed regions, then the checks for their groups and the world. Codex resumed 11 to 14 sessions at 16 regions and 10 to 55 at 64. Costs are API-equivalent USD from the provider’s usage records. Each size has its own scale.

16 regions

Follow-up, USD
Cold + follow-up, USD
Show every 16-region phase as a table
PairArmPhaseUSDWall sInput tokensCachedOutput tokensQ

64 regions

Follow-up, USD
Pair 1
$6.21
$1.62
Pair 2
$12.27
$1.46
Pair 3
$9.97
$1.73
Pair 4
$14.54
$1.62
Cold + follow-up, USD
Pair 1
$20.92
$15.40
Pair 2
$27.71
$15.86
Pair 3
$25.39
$15.70
Pair 4
$32.37
$15.27
Show every 64-region phase as a table
PairArmPhaseUSDWall sInput tokensCachedOutput tokensResumedQ
1Codex subagentsCold$14.711,178.751,786,75048,592,768346,322075
1Codex subagentsFollow-up$6.211,010.127,721,38626,377,60088,5261075
1PuppetmasterCold$13.78988.332,108,01628,939,904455,040070
1PuppetmasterFollow-up$1.62307.47,455,5857,168,51233,2061475
2Codex subagentsCold$15.441,153.553,492,50750,044,416353,524070
2Codex subagentsFollow-up$12.271,436.855,763,16753,471,616234,1973970
2PuppetmasterCold$14.411,340.835,665,37732,441,984471,519070
2PuppetmasterFollow-up$1.46288.86,794,5516,551,93631,7961470
3Codex subagentsCold$15.421,048.151,721,29148,142,208344,694070
3Codex subagentsFollow-up$9.97919.041,363,17439,157,760164,7755575
3PuppetmasterCold$13.971,235.232,747,96229,478,400448,438070
3PuppetmasterFollow-up$1.73310.66,572,5336,187,64833,7951475
4Codex subagentsCold$17.831,330.663,013,04159,127,040414,850075
4Codex subagentsFollow-up$14.541,148.359,628,58356,374,784238,9933175
4PuppetmasterCold$13.651,007.631,977,68828,946,176469,606075
4PuppetmasterFollow-up$1.62254.47,285,7427,005,69635,6372278.75

At 64 regions, pair 4 had a failed group check, so 8 more builders went back in. Follow-up wall time, just for the record: 1,161 s total for Puppetmaster against 4,514 s for Codex. Rule 6 only scores the cold build.

04 / BLIND JUDGES

Two judges. Both orders. No names.

Claude Opus 5.5 and Kimi K3.1 Pro saw the same renders of both worlds as A and B. Neither judge family built a world. Each judge saw every scene twice, once in each order. A vote counts only if it picks the same world both times. A scene counts only if both judges agree.

16 regions 24 scenes

Select a scene to read what each judge wrote.

64 regions 120 scenes

At judge size a 128 by 128 world makes every region tiny. So each phase gets 15 scenes: 3 whole-world shots and 12 quadrant close-ups.

Criterion, both judges agreePuppetmasterCodexTie
Composition38280
Geometric craft42177
Integration finish111108
Palette material00120
Readability24492

One decided scene, side by side

Pair 4 follow-up, quadrant 2-1, seed 1709. Both judges, both orders: Puppetmaster.

Codex subagents' quadrant 2-1 in pair 4 after the follow-up
Codex subagents
Puppetmaster's quadrant 2-1 in pair 4 after the follow-up, with a sandstone arch and a walled keep
Puppetmaster

Claude, Puppetmaster shown second
"World B shows more articulated landmarks, including an arch, a walled keep with corner towers, and a detailed white tower."

Kimi, Puppetmaster shown first
"World A has more distinct landmark silhouettes, richer biome transitions, and a clearer connected path network."

05 / INSIDE THE WORK

Each piece has a boundary.

The 16-region world: sixteen regions, eight landmark types. Select a region to fly both cameras to it. Gold outlines mark the three regions that the follow-up revised.

SELECT A REGION

A contract, a landmark, a place in the world.

PUPPETMASTER COLD RUN · 16 REGIONS · PAIR 4

Where the cold wall time goes

Against pair 4 of an earlier run: same Puppetmaster 1.48.0, same graph, older lab with per-worker guard checks. This study removed those checks. The comparison does not isolate one cause.

06 / HOW WE GOT HERE

We took the crew tax apart.

A crew pays a fixed price per worker: start-up, a cold read of the task, and review. Each release below went after one part of that price. The replica numbers are cold runs on one machine with no lab wrapper. Do not mix them with the study numbers.

    07 / WHERE THE LINE IS

    The judgment, in your hands.

    Puppetmaster and Marionette do not fan out because a prompt sounds big. A sizing gate reads the plan and the measured pace, then decides. This is that gate: the same decision function as puppetmaster/sizing.py, checked against it on 2,611 cases. Move the sliders.

    We measured the line. Solo against both crews · 4, 16, 64 regions

    One agent alone, Puppetmaster and Codex subagents built the same world at three sizes. Three runs per arm per size, gpt-6.1-sol on high everywhere. Two blind judges compared each solo world with each crew world, scene by scene, in both orders. The rule was locked before run one: a crew wins a size only with more scene wins, an exact binomial p under 0.05, and a majority in at least 4 of 6 comparisons.

    Measure4 regions16 regions64 regions
    Judges’ pick primarySolo holdsCrew 21, solo 16, 53 ties. 3 of 6 comparisons. p = 0.26.CrewCrew 38, solo 7, 45 ties. 5 of 6 comparisons. p = 1.6 × 10⁻⁶.CrewCrew 43, solo 0, 47 ties. 6 of 6 comparisons. p = 1.1 × 10⁻¹³.
    Puppetmaster against soloEven11 scenes to 11, 23 ties.Wins25 scenes to 0, 20 ties. 3 of 3 runs. p = 3.0 × 10⁻⁸.Wins27 scenes to 0, 18 ties. 3 of 3 runs. p = 7.5 × 10⁻⁹.
    Codex subagents against soloNot shown10 scenes to 5, 30 ties. p = 0.15.Not shown13 scenes to 7, 25 ties. p = 0.13.Wins16 scenes to 0, 29 ties. 3 of 3 runs. p = 1.5 × 10⁻⁵.
    Quality floorPassAll 9 worlds passed every structure check and Q70.PassAll 9 worlds passed every structure check and Q70.PassAll 9 worlds passed every structure check and Q70.
    Mean cost solo · PM · Codex$0.45 · $2.21 · $1.87$0.52 · $4.32 · $5.53$0.64 · $14.05 · $15.19
    Median wall solo · PM · Codex305 s · 619 s · 383 s286 s · 579 s · 418 s498 s · 1,235 s · 1,153 s

    Solo never failed. It passed every check at every size, and it was the fastest and cheapest arm. The judges still preferred a crew from 16 regions up. At 16 regions, Puppetmaster won 25 scenes to 0 against solo. Codex subagents won 13 to 7 and did not clear the rule.

    Then the crews, head to head. Puppetmaster against Codex subagents · same 18 worlds

    We took the 18 crew worlds above and asked the same two blind judges a new question: which crew built the better world? Same scenes, both orders, every vote fresh. We said in advance that this result goes on this page whichever way it went. The rule was locked before the first vote: a crew is preferred only with more scene wins over all three sizes, an exact two-sided binomial p under 0.05, and a majority in at least 6 of 9 comparisons.

    Measure4 regions16 regions64 regions
    Judges’ pickNot shownCodex subagents 17 scenes, Puppetmaster 8, 20 ties. p = 0.11.Puppetmaster16 scenes to 2, 27 ties. p = 1.3 × 10⁻³.Puppetmaster15 scenes to 1, 29 ties. p = 5.2 × 10⁻⁴.
    Runs won PM · Codex · even1 · 2 · 03 · 0 · 02 · 0 · 1
    What the judges sawCodex wins: windmills with attached sails, readable paths, less crowding.Puppetmaster wins: keeps with open interiors, walls and corner towers, roads and water that join.Puppetmaster wins: fuller landmarks, terraced pyramids, detailed keeps, readable crossings.

    The locked test, over all three sizes: Puppetmaster preferred. Puppetmaster 39 scenes, Codex subagents 20, 76 ties. p = 0.018. Six of nine runs favored Puppetmaster, two favored Codex subagents, one was even. From 16 regions up, Codex subagents won 3 of 90 scenes. At 4 regions, Codex subagents led 17 to 8. That did not pass the test, and we do not hide it: at 4 regions a crew has no quality case over solo either. A tie does not prove the crews are equal. Judge cash: $7.69 for 541 calls.

    The gate, live.

    Below 4 independent units

    Stay solo. One session that holds the whole problem beats any crew. A crew pays its start-up price on every worker.

    Measure, then decide

    The pilot starts alone. After the first finished units, the gate projects solo time against a handoff. It hands off only when the crew finishes sooner.

    Many substantial units

    At 16 or more independent units that each take real time, the gate hands off as soon as a crew projects to finish sooner. With a calibrated pace per unit, it starts with a crew.

    The measured line agrees with the gate’s default of 16 independent units, so the default stays. Solo pace per region was mostly fixed start-up work (76 s per region at 4 regions, 18 s at 16), so the gate still measures pace live and does not take a fixed value from one task.

    08 / LIMITS

    What we publish with the claim.

    Held equal

    gpt-6.1-sol on high in every worker, both sides. Inside each study: the same world, plan, cold request and follow-up request, and the same work budget (1,800 s at 16 regions, 3,600 s at 64). No routing, no effort lanes, no cheaper models.

    Changed

    Only the crew. Codex: the CLI’s built-in subagents. Puppetmaster: one flow graph with per-group builders, checks and review. puppetmaster-ai 1.48.0 at 16 regions. 1.49.0 at 64, with 8 column groups and one builder per region.

    Scope

    One world family, one lab, four pairs per size. At 64 regions, one quota stop: the two hit attempts reran once on the same pin, and the first tries stay out of the sums. The solo study: three runs per arm at 4, 16 and 64 regions with puppetmaster-ai 1.49.0. Sizes 5 to 15 not tested. Eight of its 64-region worlds come from the 64-region study and were judged there before. Every vote in the solo study is fresh. The crew head-to-head used the solo study’s 18 crew worlds, with fresh votes and no new builds.

    Money

    Builder costs are API-equivalent prices for the recorded tokens. The two blind judges cost $3.83 in real OpenRouter cash at 16 regions, $7.92 at 64, $15.35 for the solo study (1,080 calls), and $7.69 for the crew head-to-head (541 calls).

    Read the paper 16 regions (JSON)64 regions (JSON)Solo line (JSON)Crews head to head (JSON)Puppetmaster ↗