Five Parallel Agents, Zero Commits

A round of parallel coding agents on a real GPU codebase: all five died before committing, and the work was salvaged by hand — with the week's best speedup.

Summary

We run parallel coding agents in git worktrees against a real codebase — a Metal inference engine, where every change has to stay numerically identical to a reference implementation. The first round merged three branches cleanly. The second round: all five agents died before committing anything. Two ran out of API credits mid-task, three were killed by a stream watchdog. Their worktrees still contained the work, and salvaged by hand it included the largest single speedup of the week — a mixture-of-experts prefill path going from 193 to 369 tok/s with identical output. The failure was entirely in the harness around the agents, not in what the agents produced.

What actually happened

Five agents, five branches, five worktrees, each with its own build output directory on a shared workstation so they don't overwrite each other's binaries. Long tasks — kernel work, hours each.

All five died early:

• two out of API credits. A budget that was fine for a single agent is not fine for five running concurrently for hours.
• three on a stream watchdog. A long-running agent that goes quiet while a compile or a benchmark runs looks, to a watchdog, exactly like a hung one.

Nothing was committed. The coordinating session had moved on. The worktrees sat on disk with uncommitted changes in them, which is the only reason any of it survived.

The three failures worth designing around

1. "Finished" is not "committed." Our round-one process assumed an agent that reports success has landed its work on a branch. Round two had no successes to report — but the same gap exists on the happy path, because an agent that finishes its edits and then dies during the commit is indistinguishable from one that never started. The coordinator has to inspect the trees, not read the reports. git -C <worktree> status --porcelain across every worktree, at the end of every round, unconditionally.

2. Agent claims need testing, not trusting. This is not about honesty; it's about scope. An agent optimizing one kernel in round one asserted the kernel "should be around 10 µs." We measured: the floor for merely touching that layer's state is 16 µs, and the existing kernel was about 7 µs above the floor. The target was below what the memory traffic permits. An agent optimizing a component sees the component, not the floor, and will happily spend hours chasing a number that isn't reachable. Every claim in this project had to clear a bit-identical output check or a back-to-back interleaved measurement before it counted.

3. Concurrency multiplies quota, watchdogs, and workspace collisions. Five agents is five times the token burn against one budget, five long-quiet processes against one watchdog policy, and five build systems against one output directory. Each needs an explicit decision. We only had one of the three.

The git-worktree traps

Two specific ones cost us time and are easy to hit.

A linked worktree points at the original clone, not at a copy of it. If the repository directory is copied somewhere else, the worktree references still resolve to the original checkout, and git merge <agent-branch> in the copy finds nothing to merge — silently, because from git's perspective there's simply no such branch there. Copy the files out of the worktree instead of merging whenever the repo has been duplicated.

git add -A will commit the worktree directory as an embedded repository. It did, once. git reset HEAD <worktree dir> before committing, or gitignore the whole directory, which is what we ended up doing.

What the dead agents had produced

The salvage is the reason this isn't a "parallel agents don't work" post:

• A fast verify GEMM for speculative decoding. An 8-row verify went from 2.5 decode steps to 1.8; a 16-row verify from 4.7 to 2.7. Exposed behind a recipe flag so the exact path stays available.
• A grouped expert GEMM for the mixture-of-experts model — tiles of 32 slots of a single expert, with two projections issued in one dispatch via a grid plane. Prefill went 193 → 369 tok/s, with identical tokens. Best result of the week.
• A position-indexed ring buffer for the attention compressor state, bit-identical, which turned out to be the prerequisite for speculative decoding on that model.
• Serving hooks and a chunked reference implementation in the test suite.

None of that was committed by the agent that wrote it.

And round one, honestly

Round one merged three branches and is the more interesting data point for whether this is worth doing, because it succeeded and still mostly didn't pay:

The largest merge fused a chunk of the linear-attention path — dispatches per token dropped from 770 to 674 autoregressive and 795 to 693 speculative, bit-identical. Net effect on wall clock: under 1%. Fewer dispatches is not faster when you were never dispatch-bound.

What that round was actually worth:

• It found and fixed an out-of-bounds write in a convolution kernel on prefill chunks — a real latent bug, in code nobody was looking at.
• It produced two durable negative results: the state-touch floor above, and that whole-head fused attention kernels lose as soon as more than one token is in flight.

So the honest ledger for round one is: one bug fixed, two dead ends closed, ~1% of speed. That's a normal week of engineering, which is a fair thing for a round of agents to be worth — but it is not the 5× that "five agents in parallel" suggests, and we'd have been disappointed if we'd been expecting throughput rather than coverage.

What we changed

• Sweep every worktree for uncommitted work at the end of a round, before reading a single report.
• Budget and watchdog policy set per round, sized for N agents, not for one.
• Each agent gets its own build output directory. Non-negotiable on a shared machine.
• Every claim clears a bit-identical check or an interleaved A/B/A before merge. On a shared box, absolute throughput drifts ±20–30% with unrelated processes, so a "before" number from yesterday is not a baseline.
• Prune merged branches and worktrees at the end of every round, or the next round starts in a swamp.

Five parallel agents and zero commits is a tooling problem, not a people problem — the kind of thing an on-site package sorts out.

Key takeaways

1. Sweep the worktrees, don't read the reports. All five agents died with nothing committed, and every one of them had usable work on disk.
2. An agent that reports success may not have committed. Verify with git status, not with prose.
3. Concurrency multiplies your quota, your watchdog exposure, and your workspace collisions. Two of ours ran out of credits; three were killed for being quiet during a compile.
4. Test agent claims against a physical floor. One targeted 10 µs on a kernel whose pure memory traffic costs 16.
5. A linked worktree points at the original clone. Merge finds nothing in a copied repo — copy the files out instead.
6. Fewer dispatches is not faster if you were never dispatch-bound. A clean bit-identical fusion that cut dispatches 12% moved wall clock under 1%.
7. The right expectation for a parallel round is coverage, not throughput — a latent out-of-bounds write and two closed dead ends is a good outcome, and it isn't a speedup.