LABS · PAPER · 16-REGION STUDY

Same model, different crew

Cary Palmer · 8 November 2026 · Puppetmaster 1.48.0 · Sol 6.1 high

Abstract

We gave two orchestrators the same model, the same effort, the same task and the same budget. One arm was the Codex CLI with its own built-in subagents. We call this arm native. The other arm was Puppetmaster: one flow graph with per-region builders, checks, group reviews and resumable worker sessions.

The task was a 16-region voxel world, then a follow-up request that changed three regions. We froze six decision rules by SHA-256 before the first run. We ran four complete pairs. Two blind judges from other model families compared the worlds in both orders.

Puppetmaster cost less in every pair and every phase. The follow-up median was $0.738 against $2.600, 71.6% lower (exact Mann-Whitney p = 0.014). The blind judges decided 9 scenes for Puppetmaster, 0 for native, and 15 ties (exact one-sided p = 0.002). Cold wall time showed no difference by the frozen rule.

Follow-up cost−71.6%
Full task cost−35.3%
Blind scenes9–0
Pairs cheaper4/4

“Puppetmaster costs less and makes better worlds than native subagents on this task, with the same model and effort.”

That is the exact claim text that the frozen rules allow when rules 1, 2, 3 and 5 hold. We do not widen it. It covers one task, one lab, four pairs, Sol 6.1 at high effort and Puppetmaster 1.48.0.

  1. The question
  2. Design
  3. Rules, written first
  4. The quality instrument
  5. Results
  6. Where the follow-up saving comes from
  7. Cold wall time and the lab
  8. Predictions against the result
  9. The product changes behind this run
  10. Where the line is
  11. Limits and disclosures
  12. Data

1. The question

A strong model alone does small work best. One session holds the whole problem. Every extra worker pays a fixed start: its own context, its own reading, its own checks. The pilot then pays again to join the results.

Past some size, one session falls off. The context fills, and the work runs past its time. Modern agent CLIs answer with native subagents. So the real question for an orchestrator is narrow:

Past the point where one session falls off, does a durable orchestrator beat the native subagents of the same model? Same model, same effort, no cheaper workers, no routing. Only the crew changes.

This study tests that on one task. An earlier run gave a cost lead and a quality tie. Its quality rubric was saturated, and its lab completed 2 of 5 pairs. This study fixed those two causes and ran the same treatment again.

2. Design

The task

A 64 × 64 voxel world on a 4 × 4 grid of regions. Each region has a contract: a biome and a landmark (windmill, watchtower, temple, lodge, arch, shrine, lighthouse or keep). The cold request builds and repairs all 16 regions to the contracts. The follow-up request then revises three regions: region_001 (watchtower), region_005 (shrine) and region_015 (keep).

The two arms

NativePuppetmaster
OrchestratorThe agent CLI and its own subagentsOne flow graph, puppetmaster-ai==1.48.0
ShapeThe model plans and spawns subagentsFour group items, one builder per region inside each group, a check and a render per group, a group review, then a world judge
ModelSol 6.1 (gpt-6.1-sol), high effort, in every worker of both arms
Budget1800 s work, 120 s cleanup, per phase
InputsThe same task files, contracts, cold request and follow-up request
RoutingNone. No escalate_model, no effort lanes, no cheaper models

The plan

What the lab could refuse

The lab could refuse a phase for five causes only: a wrong model, effort, binary or Puppetmaster version; missing usage; no settled workspace; contaminated inputs; or a phase that never started. Every other lab check recorded a flag, and the phase counted. An actor that ran past its budget was an actor failure, not a refusal.

3. Rules, written first

We wrote six rules and their order, then froze the declaration by SHA-256 before rep 1. Each rule gives its own verdict. A later rule cannot cancel an earlier one. All rules use the four complete pairs only.

  1. Quality gate. It fails if Puppetmaster qualifies less often than native, or if native wins a significant scene-vote majority. If it fails, no win claim of any kind follows.
  2. Follow-up cost (primary). Puppetmaster lower in 3 of 4 pairs or more, exact one-sided Mann-Whitney U ≤ 1 (p ≤ 0.05), and a median at least 30% lower.
  3. Full task cost. Cold plus follow-up lower in 3 of 4 pairs or more, and in the sum.
  4. Cold cost. Lower in 3 of 4 pairs or more, and in the sum.
  5. Quality win. Puppetmaster wins the scene-vote majority in 3 of 4 pairs or more, with an exact one-sided binomial p ≤ 0.05 on decided scenes. Ties are not decided votes.
  6. Wall. Puppetmaster cold wall lower in 3 of 4 pairs or more, and in the sum. A wall loss does not block rules 2 to 5.

The declaration also fixed the only public claim text for each outcome.

4. The quality instrument

Qualification

A blind rubric grade scores each world from 0 to 100 on five dimensions: composition, geometric craft, integration and finish, palette and material, and readability. A world qualifies at Q70. In the earlier run every cold run in both arms got the same scores, so this grade is a gate only. It cannot show which world is better.

Blind pairwise preference

Calibration before rep 1

TestPass barResult
Identity: a world against a byte copy80% or more ties3 of 3 scenes tie
Gross defect: one group region removed90% or more pick the intact world3 of 3 scenes pick intact
Mild defect: 10% of one region's blocks removedDescriptive only3 of 3 scenes tie

The instrument passed. The mild-defect row also shows its resolution: it does not see a small planted defect. A decided scene therefore marks a difference the two judges both see in both orders.

We then ran the frozen instrument on the earlier run’s worlds as an end-to-end test. All 15 of its scenes were ties. That result is descriptive. It did not change any rule.

5. Results

RuleVerdictMeasured
1 · Quality gatePassBoth arms qualified in 4 of 4 pairs. Native won no scene majority.
2 · Follow-up costHoldsLower in 4 of 4. Median $0.738 against $2.600, 71.6% lower. U = 0, exact p = 1/70 = 0.014.
3 · Full task costHoldsLower in 4 of 4. $19.68 against $30.45, 35.3% lower.
4 · Cold costHoldsLower in 4 of 4. $16.70 against $20.59, 18.9% lower.
5 · Quality winHoldsMajority in 4 of 4 pairs. 9 scenes Puppetmaster, 0 native, 15 ties. Exact one-sided p = 0.002.
6 · Cold wallNo differenceFaster in 2 of 4. Sum 1907.1 s against 2099.6 s.
Native subagentsPuppetmaster
Follow-up cost per pair in API-equivalent USD. Native: 2.73, 2.55, 2.65, 1.92. Puppetmaster: 0.75, 0.79, 0.72, 0.73. Pair 1$2.73$0.75 Pair 2$2.55$0.79 Pair 3$2.65$0.72 Pair 4$1.92$0.73
Figure 1. Follow-up cost per pair, API-equivalent USD at fixed list prices. Bar length is proportional to cost from a zero baseline.

Every phase

PairArmCold USDFollow-up USDCold QFollow-up QCold wall sScenes PM / native / tie
1Native5.6752.73178.7575.00546.21 / 0 / 5
Puppetmaster3.8550.74678.7575.00424.5
2Native5.0592.55278.7575.00542.42 / 0 / 4
Puppetmaster4.8790.79278.7575.00620.7
3Native5.0182.64878.7575.00439.42 / 0 / 4
Puppetmaster4.0630.71878.7575.00491.4
4Native4.8421.92278.7578.75571.74 / 0 / 2
Puppetmaster3.9020.73075.0073.75370.5

Costs are API-equivalent USD from the provider usage records at fixed list prices. Scenes count cold and follow-up together, 6 per pair.

6. Where the follow-up saving comes from

Both arms did the follow-up correctly. A per-voxel diff of each cold world against its follow-up world shows changes in regions 1, 5 and 15 only, in all eight follow-ups. The difference is in how much work each arm did to get there.

The follow-up was also faster, though no rule tests it: Puppetmaster 150.6 s to 168.4 s per phase, native 343.5 s to 403.5 s. We report this as descriptive only.

This is the structural result. A native subagent's context belongs to the parent session. A Puppetmaster worker's session belongs to its unit. When a change names three units, Puppetmaster can resume those three workers and leave the rest alone.

In the cold phase the gap is smaller. Puppetmaster read 41.0M input tokens against native 72.5M, but wrote more output (519k against 475k). Cold cost was 18.9% lower.

7. Cold wall time and the lab

Cold wall time showed no difference by rule 6. Puppetmaster was faster in pairs 1 and 4, slower in pairs 2 and 3, and 9% lower in the sum.

We predicted a loss of 1.5 to 1.9 times. The prediction came from the earlier run, which ran the same Puppetmaster version and the same graph in an older lab with per-worker guard checks. This study removed those checks.

Puppetmaster coldAdmission to flow sFlow to map sGroup map sWorld judge sFailed reviewsFull wall s
Earlier run, pair 4137.942.0678.0151.021039.8
This study, pair 111.81.0279.0125.01424.5
This study, pair 29.71.0512.088.02620.7
This study, pair 314.61.0366.0100.01491.4
This study, pair 411.61.0265.081.00370.5

The median builder dispatch, from node start to the first model turn, was 0.5 s to 1.1 s in this study and 85.7 s in pair 4 of the earlier run. Most of that wall gap was the lab, not the product. The comparison does not isolate one cause, and it is descriptive. Rule 6 compares the two arms inside this study only.

Failed reviews still cost time. Pair 2 had two failed group reviews and the longest map. The critical path of a map is its slowest item.

8. Predictions against the result

We wrote predictions into the declaration so the result could prove them wrong.

RulePredictionResult
1PassesPasses
2Holds. Puppetmaster $0.7 to $1.4, native $3.0 to $3.8Holds. $0.718 to $0.792 against $1.922 to $2.731
3Holds, 25% to 40% lowerHolds, 35.3% lower
4Does not hold, gap within 15%Holds, 18.9% lower, 4 of 4
5Does not hold, share 45% to 65%Holds, 9 to 0
6Does not hold, 1.5 to 1.9 times slowerNo difference, sum 9% lower

Three predictions were wrong, and all three in the Puppetmaster direction.

9. The product changes behind this run

This study measures Puppetmaster 1.48.0. The releases between 15 and 24 October 2026 changed how a flow builds and repairs. We measured each on a local replica of the same world task: cold, Sol 6.1 high, no pilot, no lab guards. Replica numbers are wall time only. Do not compare them with this study’s lab numbers.

ReleaseChangeReplica cold median
1.44.0A repair sends only the change. A re-review names the changed files.n/a
1.45.0A follow-up keeps a map item when its task, inputs and outputs are the same and its review still passes.n/a
1.46.0Base graph, one builder per group (reference)570 s, 7 runs
1.46.0Builders fork one primed session492 s, 3 runs
1.46.1One builder per region inside each group392 s, 5 runs
1.47.0Codex workers run as threads in one shared host371 s, 3 runs (373 s without)
—Native subagents, same replica450 s, 3 runs

The shared host gave almost nothing on the replica, which has little start-up cost to remove. A guarded lab has much more. Per-region builders were the change that moved the replica median below native.

10. Where the line is

This study is one point past the line. It does not find the line. Puppetmaster ships the line as a sizing gate (puppetmaster/sizing.py). The gate watches the solo pace and answers one of three ways:

The Labs page runs a line-for-line port of this function. A parity script checks it against the Python on 2,611 inputs with 0 differences.

These defaults come from an earlier study on an older Puppetmaster. We did not measure them again after the changes in section 9. Treat them as provisional.

What this study adds to the judgment

What we do not know yet

A later study measured the line again with puppetmaster-ai 1.49.0. Blind judges preferred a crew over solo from 16 regions up, and preferred the Puppetmaster crew over Codex subagents from 16 regions up. That agrees with the gate’s default of 16 units. The sizing hook records when a turn starts, but not the decision or the outcome. Marionette's judgment needs both. The next step is to log each decision and its result per model, and to fit the thresholds from that data.

11. Limits and disclosures

12. Data

Cite as: Palmer, C. “Same model, different crew.” Puppetmaster Labs, 8 November 2026. https://puppetmaster.gg/labs/same-model/