LABS · PAPER · 16-REGION STUDY
Same model, different crew
Abstract
We gave two orchestrators the same model, the same effort, the same task and the same budget. One arm was the Codex CLI with its own built-in subagents. We call this arm native. The other arm was Puppetmaster: one flow graph with per-region builders, checks, group reviews and resumable worker sessions.
The task was a 16-region voxel world, then a follow-up request that changed three regions. We froze six decision rules by SHA-256 before the first run. We ran four complete pairs. Two blind judges from other model families compared the worlds in both orders.
Puppetmaster cost less in every pair and every phase. The follow-up median was $0.738 against $2.600, 71.6% lower (exact Mann-Whitney p = 0.014). The blind judges decided 9 scenes for Puppetmaster, 0 for native, and 15 ties (exact one-sided p = 0.002). Cold wall time showed no difference by the frozen rule.
“Puppetmaster costs less and makes better worlds than native subagents on this task, with the same model and effort.”
That is the exact claim text that the frozen rules allow when rules 1, 2, 3 and 5 hold. We do not widen it. It covers one task, one lab, four pairs, Sol 6.1 at high effort and Puppetmaster 1.48.0.
- The question
- Design
- Rules, written first
- The quality instrument
- Results
- Where the follow-up saving comes from
- Cold wall time and the lab
- Predictions against the result
- The product changes behind this run
- Where the line is
- Limits and disclosures
- Data
1. The question
A strong model alone does small work best. One session holds the whole problem. Every extra worker pays a fixed start: its own context, its own reading, its own checks. The pilot then pays again to join the results.
Past some size, one session falls off. The context fills, and the work runs past its time. Modern agent CLIs answer with native subagents. So the real question for an orchestrator is narrow:
Past the point where one session falls off, does a durable orchestrator beat the native subagents of the same model? Same model, same effort, no cheaper workers, no routing. Only the crew changes.
This study tests that on one task. An earlier run gave a cost lead and a quality tie. Its quality rubric was saturated, and its lab completed 2 of 5 pairs. This study fixed those two causes and ran the same treatment again.
2. Design
The task
A 64 × 64 voxel world on a 4 × 4 grid of regions. Each region has a contract: a biome and a landmark (windmill, watchtower, temple, lodge, arch, shrine, lighthouse or keep). The cold request builds and repairs all 16 regions to the contracts. The follow-up request then revises three regions: region_001 (watchtower), region_005 (shrine) and region_015 (keep).
The two arms
| Native | Puppetmaster | |
|---|---|---|
| Orchestrator | The agent CLI and its own subagents | One flow graph, puppetmaster-ai==1.48.0 |
| Shape | The model plans and spawns subagents | Four group items, one builder per region inside each group, a check and a render per group, a group review, then a world judge |
| Model | Sol 6.1 (gpt-6.1-sol), high effort, in every worker of both arms | |
| Budget | 1800 s work, 120 s cleanup, per phase | |
| Inputs | The same task files, contracts, cold request and follow-up request | |
| Routing | None. No escalate_model, no effort lanes, no cheaper models | |
The plan
- Each pair has four phases: native cold, native follow-up, Puppetmaster cold, Puppetmaster follow-up.
- Each follow-up starts from its own arm's settled cold workspace and resumes its own cold session history.
- The arm order alternates in each rep. The first four complete pairs count. The plan allowed six reps at most.
- A dress rehearsal ran first. Its numbers are not in the result.
What the lab could refuse
The lab could refuse a phase for five causes only: a wrong model, effort, binary or Puppetmaster version; missing usage; no settled workspace; contaminated inputs; or a phase that never started. Every other lab check recorded a flag, and the phase counted. An actor that ran past its budget was an actor failure, not a refusal.
3. Rules, written first
We wrote six rules and their order, then froze the declaration by SHA-256 before rep 1. Each rule gives its own verdict. A later rule cannot cancel an earlier one. All rules use the four complete pairs only.
- Quality gate. It fails if Puppetmaster qualifies less often than native, or if native wins a significant scene-vote majority. If it fails, no win claim of any kind follows.
- Follow-up cost (primary). Puppetmaster lower in 3 of 4 pairs or more, exact one-sided Mann-Whitney U ≤ 1 (p ≤ 0.05), and a median at least 30% lower.
- Full task cost. Cold plus follow-up lower in 3 of 4 pairs or more, and in the sum.
- Cold cost. Lower in 3 of 4 pairs or more, and in the sum.
- Quality win. Puppetmaster wins the scene-vote majority in 3 of 4 pairs or more, with an exact one-sided binomial p ≤ 0.05 on decided scenes. Ties are not decided votes.
- Wall. Puppetmaster cold wall lower in 3 of 4 pairs or more, and in the sum. A wall loss does not block rules 2 to 5.
The declaration also fixed the only public claim text for each outcome.
4. The quality instrument
Qualification
A blind rubric grade scores each world from 0 to 100 on five dimensions: composition, geometric craft, integration and finish, palette and material, and readability. A world qualifies at Q70. In the earlier run every cold run in both arms got the same scores, so this grade is a gate only. It cannot show which world is better.
Blind pairwise preference
- Every scene of both worlds is rendered from the same fixed cameras, re-encoded to one format and size, and stripped of paths and metadata.
- A judge sees the request text, the five criteria, and the two worlds as A and B. It answers in JSON: A, B or tie, a pick per criterion, and one reason.
- Two judges from model families that built nothing:
anthropic/claude-opus-5.5andmoonshotai/kimi-k3.1-pro. Temperature 0, one frozen prompt. - Each scene goes to each judge in both orders. A judge vote counts only if the judge picks the same world in both orders. Otherwise it is a tie.
- A scene counts for a world only if both judges vote for that world. Otherwise the scene is a tie.
Calibration before rep 1
| Test | Pass bar | Result |
|---|---|---|
| Identity: a world against a byte copy | 80% or more ties | 3 of 3 scenes tie |
| Gross defect: one group region removed | 90% or more pick the intact world | 3 of 3 scenes pick intact |
| Mild defect: 10% of one region's blocks removed | Descriptive only | 3 of 3 scenes tie |
The instrument passed. The mild-defect row also shows its resolution: it does not see a small planted defect. A decided scene therefore marks a difference the two judges both see in both orders.
We then ran the frozen instrument on the earlier run’s worlds as an end-to-end test. All 15 of its scenes were ties. That result is descriptive. It did not change any rule.
5. Results
| Rule | Verdict | Measured |
|---|---|---|
| 1 · Quality gate | Pass | Both arms qualified in 4 of 4 pairs. Native won no scene majority. |
| 2 · Follow-up cost | Holds | Lower in 4 of 4. Median $0.738 against $2.600, 71.6% lower. U = 0, exact p = 1/70 = 0.014. |
| 3 · Full task cost | Holds | Lower in 4 of 4. $19.68 against $30.45, 35.3% lower. |
| 4 · Cold cost | Holds | Lower in 4 of 4. $16.70 against $20.59, 18.9% lower. |
| 5 · Quality win | Holds | Majority in 4 of 4 pairs. 9 scenes Puppetmaster, 0 native, 15 ties. Exact one-sided p = 0.002. |
| 6 · Cold wall | No difference | Faster in 2 of 4. Sum 1907.1 s against 2099.6 s. |
Every phase
| Pair | Arm | Cold USD | Follow-up USD | Cold Q | Follow-up Q | Cold wall s | Scenes PM / native / tie |
|---|---|---|---|---|---|---|---|
| 1 | Native | 5.675 | 2.731 | 78.75 | 75.00 | 546.2 | 1 / 0 / 5 |
| Puppetmaster | 3.855 | 0.746 | 78.75 | 75.00 | 424.5 | ||
| 2 | Native | 5.059 | 2.552 | 78.75 | 75.00 | 542.4 | 2 / 0 / 4 |
| Puppetmaster | 4.879 | 0.792 | 78.75 | 75.00 | 620.7 | ||
| 3 | Native | 5.018 | 2.648 | 78.75 | 75.00 | 439.4 | 2 / 0 / 4 |
| Puppetmaster | 4.063 | 0.718 | 78.75 | 75.00 | 491.4 | ||
| 4 | Native | 4.842 | 1.922 | 78.75 | 78.75 | 571.7 | 4 / 0 / 2 |
| Puppetmaster | 3.902 | 0.730 | 75.00 | 73.75 | 370.5 |
Costs are API-equivalent USD from the provider usage records at fixed list prices. Scenes count cold and follow-up together, 6 per pair.
6. Where the follow-up saving comes from
Both arms did the follow-up correctly. A per-voxel diff of each cold world against its follow-up world shows changes in regions 1, 5 and 15 only, in all eight follow-ups. The difference is in how much work each arm did to get there.
- Puppetmaster ran less of the graph. In all four follow-ups the flow ran three region builders (
region_001,region_005,region_015), the reviews for the two groups that hold them, and the world judge. The other two groups did not run. No review failed. - It resumed fewer sessions. The Puppetmaster follow-up resumed 7 sessions in every pair: the pilot, three builders, two group reviewers and the world judge. The native follow-up resumed 11 to 14 sessions.
- Each resumed session read less. Input tokens in the follow-up: native 46.05M over four pairs, Puppetmaster 13.27M, 71.2% fewer. Cache share was the same in both arms (95.8% and 95.3%), so the saving is volume, not caching.
- It wrote less. Follow-up output tokens: native 34.6k to 45.9k per phase, Puppetmaster 11.4k to 13.0k.
The follow-up was also faster, though no rule tests it: Puppetmaster 150.6 s to 168.4 s per phase, native 343.5 s to 403.5 s. We report this as descriptive only.
This is the structural result. A native subagent's context belongs to the parent session. A Puppetmaster worker's session belongs to its unit. When a change names three units, Puppetmaster can resume those three workers and leave the rest alone.
In the cold phase the gap is smaller. Puppetmaster read 41.0M input tokens against native 72.5M, but wrote more output (519k against 475k). Cold cost was 18.9% lower.
7. Cold wall time and the lab
Cold wall time showed no difference by rule 6. Puppetmaster was faster in pairs 1 and 4, slower in pairs 2 and 3, and 9% lower in the sum.
We predicted a loss of 1.5 to 1.9 times. The prediction came from the earlier run, which ran the same Puppetmaster version and the same graph in an older lab with per-worker guard checks. This study removed those checks.
| Puppetmaster cold | Admission to flow s | Flow to map s | Group map s | World judge s | Failed reviews | Full wall s |
|---|---|---|---|---|---|---|
| Earlier run, pair 4 | 137.9 | 42.0 | 678.0 | 151.0 | 2 | 1039.8 |
| This study, pair 1 | 11.8 | 1.0 | 279.0 | 125.0 | 1 | 424.5 |
| This study, pair 2 | 9.7 | 1.0 | 512.0 | 88.0 | 2 | 620.7 |
| This study, pair 3 | 14.6 | 1.0 | 366.0 | 100.0 | 1 | 491.4 |
| This study, pair 4 | 11.6 | 1.0 | 265.0 | 81.0 | 0 | 370.5 |
The median builder dispatch, from node start to the first model turn, was 0.5 s to 1.1 s in this study and 85.7 s in pair 4 of the earlier run. Most of that wall gap was the lab, not the product. The comparison does not isolate one cause, and it is descriptive. Rule 6 compares the two arms inside this study only.
Failed reviews still cost time. Pair 2 had two failed group reviews and the longest map. The critical path of a map is its slowest item.
8. Predictions against the result
We wrote predictions into the declaration so the result could prove them wrong.
| Rule | Prediction | Result |
|---|---|---|
| 1 | Passes | Passes |
| 2 | Holds. Puppetmaster $0.7 to $1.4, native $3.0 to $3.8 | Holds. $0.718 to $0.792 against $1.922 to $2.731 |
| 3 | Holds, 25% to 40% lower | Holds, 35.3% lower |
| 4 | Does not hold, gap within 15% | Holds, 18.9% lower, 4 of 4 |
| 5 | Does not hold, share 45% to 65% | Holds, 9 to 0 |
| 6 | Does not hold, 1.5 to 1.9 times slower | No difference, sum 9% lower |
Three predictions were wrong, and all three in the Puppetmaster direction.
9. The product changes behind this run
This study measures Puppetmaster 1.48.0. The releases between 15 and 24 October 2026 changed how a flow builds and repairs. We measured each on a local replica of the same world task: cold, Sol 6.1 high, no pilot, no lab guards. Replica numbers are wall time only. Do not compare them with this study’s lab numbers.
| Release | Change | Replica cold median |
|---|---|---|
| 1.44.0 | A repair sends only the change. A re-review names the changed files. | n/a |
| 1.45.0 | A follow-up keeps a map item when its task, inputs and outputs are the same and its review still passes. | n/a |
| 1.46.0 | Base graph, one builder per group (reference) | 570 s, 7 runs |
| 1.46.0 | Builders fork one primed session | 492 s, 3 runs |
| 1.46.1 | One builder per region inside each group | 392 s, 5 runs |
| 1.47.0 | Codex workers run as threads in one shared host | 371 s, 3 runs (373 s without) |
| — | Native subagents, same replica | 450 s, 3 runs |
The shared host gave almost nothing on the replica, which has little start-up cost to remove. A guarded lab has much more. Per-region builders were the change that moved the replica median below native.
10. Where the line is
This study is one point past the line. It does not find the line. Puppetmaster ships the line as a sizing gate (puppetmaster/sizing.py). The gate watches the solo pace and answers one of three ways:
- Stay solo when fewer than 4 independent units remain, when no unit is finished yet, when 80% of the units are done, or when solo work is projected to finish inside its horizon.
- Hand off when solo work is projected to run past 1200 s or 60% of the context window, and a crew finishes the rest sooner. A crew pays about 90 s to start. With 16 or more independent units of 20 s or more each, it hands off as soon as a crew is projected to finish sooner.
- Delegate up front only with a calibrated per-unit time and 16 or more substantial independent units.
The Labs page runs a line-for-line port of this function. A parity script checks it against the Python on 2,611 inputs with 0 differences.
These defaults come from an earlier study on an older Puppetmaster. We did not measure them again after the changes in section 9. Treat them as provisional.
What this study adds to the judgment
- Count future revisions. Most of the saving was in the follow-up. A crew with per-unit sessions is worth more when the work will change later. The gate does not count that today.
- The crew tax is mostly start-up and review. In this lab, flow admission fell to about 12 s and builder dispatch to about 1 s. Failed reviews and the slowest map item set the wall time.
- Same model is enough. This study used no cheaper workers. The saving came from structure, not from price.
What we do not know yet
A later study measured the line again with puppetmaster-ai 1.49.0. Blind judges preferred a crew over solo from 16 regions up, and preferred the Puppetmaster crew over Codex subagents from 16 regions up. That agrees with the gate’s default of 16 units. The sizing hook records when a turn starts, but not the decision or the outcome. Marionette's judgment needs both. The next step is to log each decision and its result per model, and to fit the thresholds from that data.
11. Limits and disclosures
- Scope. One task, one lab, four pairs, one model at one effort, one Puppetmaster version. A second study repeats the comparison at 64 regions. The same rules gave the same verdicts there.
- Judge position bias. Kimi picked the world it saw first in 39 of 48 calls. The both-orders rule turned that bias into ties. Kimi never gave an order-consistent pick for native.
- One judge limits the count. Claude gave an order-consistent Puppetmaster pick in 23 of 24 scenes and a tie in 1. Kimi agreed in 9. Kimi set the number of decided scenes.
- The two quality instruments disagree on pair 4. The rubric scored native higher (78.75 and 78.75 against 75.00 and 73.75). The preference judges picked Puppetmaster in 4 scenes of that pair. Both arms qualified at Q70 in every phase.
- Resolution. The instrument did not see a mild planted defect, and it found all scenes from the earlier run tied.
- Pins. Pair 1 ran on pin 1. Pairs 2 to 4 ran on pin 2 after a provider quota stop, and each used the authorized cold retry for both arms. All four phase types had the same launch hash on both pins.
- Refusals. Ten original cold attempts were refused under lab cause 2 (missing usage). They stay in the record and are not in the result.
- Money. Actor costs are API-equivalent prices for the recorded tokens, not cash. The two judges cost $3.83 in OpenRouter cash for the whole study.
- Authorship. We build Puppetmaster. A product owner froze the rules, the predictions and the claim text before the first run, and a separate lab operator ran the cell.
12. Data
- r208.json: every phase (cost, tokens, wall, Q, world hash), every judge vote and reason, the Puppetmaster cold node timings, and the rule results.
- The 16 delivered worlds, one file per arm, phase and pair, under
/labs/data/. The study page renders them. - Final report SHA-256
139ab93b6dd8ecc0151082455613a87ba1897584f2df67ffdc4bb199fd7b62ee. Declaration SHA-2562db71a815b2eb48fb311e449bf2928f368fdf60dc87e33950c346a3793b37e95.
Cite as: Palmer, C. “Same model, different crew.” Puppetmaster Labs, 8 November 2026. https://puppetmaster.gg/labs/same-model/
