אִם יִרְצֶה הַשֵּׁם
Nate Jones published a one-minute test this week for whether a task wants a chat, one agent, a team, or nothing. The wedge underneath it, checking has to be cheaper than producing, is the same wedge that explains why AI-generated code keeps shipping through green CI runs and into the 2026 incident corpus. His test asks whether you can verify the answer. It does not ask whether you built a verifier for the layer where the attack actually lands.
Nate's source is the Stanford "Large Language Monkeys" paper: a cheap coder model given 250 attempts per SWE-bench Lite bug hit 56 percent, beating a frontier model's single-shot 43. The half of that paper he flags and most citations skip: pushed to 10,000 attempts, coverage (a correct answer exists somewhere in the pile) climbed past 95 percent, while selection (anyone can find it without a mechanical checker) stalled around 100 attempts, flat, regardless of method.
That stall is the whole story. Spend buys results only where checking an answer costs less than producing one. Everywhere else it just buys a bigger pile you cannot search.
The Estimate His Framework Skips
Nate's four estimates, size, independence, separation, checkability, are aimed at the task. Run the same test on the artifact a coding agent hands you and a fifth question shows up, one his framework has no slot for: checkable against what?
| Layer | Checker exists | What ships anyway |
|---|---|---|
| Logic, interface behavior | Yes, the test suite | Correct output, wrong assumptions |
| Dependency provenance | Rarely | Typosquats, hijacked maintainer accounts, transitive pulls nobody reviewed |
| Permission scope | Rarely | Post-install scripts, CI tokens the agent never needed |
| Agent context files | Almost never | Instructions the agent reads that the reviewer never sees, same trick as TrapDoor |
A green test suite answers row one. Mini Shai-Hulud and TrapDoor both shipped through pipelines where row one was clean. Nobody had built a checker for rows two through four, so no amount of agent spend on those pipelines could have helped. The checkability estimate was never run on the layer that failed.
Adding Agents Does Not Fix an Unchecked Layer
If checkability is the gate, more agents cannot force it open, and a December 2025 Google/MIT paper measured exactly that failure shape. Kim et al., "Towards a Science of Scaling Agent Systems," ran 180 controlled configurations across five architectures and three model families. Two numbers matter here.
Tool-heavy tasks lose to coordination overhead. Splitting the token budget across agents starves each one of the context a real tool-use loop needs. The paper's tool-heavy configurations ran at a 2 to 6x token penalty against a single agent doing the same work, before counting the coordination messages.
Uncontained agents amplify errors instead of catching them. Independent, unmanaged multi-agent setups amplified error rates 17.2x over a single-agent baseline. Centralized review, one agent whose entire job is checking the others, contained that to 4.4x. The paper's own threshold: once a single agent already clears 45 percent on a task, adding agents past that point tends to make it worse.
Read against the checkability wedge, both numbers say the same thing. A team of agents is a way to spend more tokens. It is not a way to manufacture a checker where none exists. If nothing catches a hijacked dependency today, running three agents over the same pull request does not create that catch. It creates three unverified opinions instead of one.
Run This Before You Trust a Green Build
Ask what the checker actually covers. Test suites check the function the agent was asked to write. They were not built to check the fifteen transitive dependencies it pulled in to write it. Write down the boundary of what CI verifies before you extend how much you trust past that boundary.
Do not count a same-model reviewer as separation. An agent grading its own generation, or a sibling instance with the same training and the same blind spots, is not a second mind. It is the auditor keeping the books again, run twice. Real separation needs a reviewer with a different failure surface: a different model family, a mechanical scanner, or a human who did not write the change.
Match agent count to where checking is cheap, not where the work is large. The Google/MIT numbers say coordination overhead eats tool-heavy, sequential work fastest, which is exactly the shape of a real coding task: read the repo, pull dependencies, run tools, write code. Adding agents there multiplies cost before it multiplies coverage. Spend the added agents on the row of the table above that has no checker yet, not on the row that already has one.
Build the missing checker before you scale the swarm. Provenance, permission scope, and context-file integrity are checkable. Nobody has made them as cheap to check as a unit test, so nobody's agent spend goes there. That gap, not agent capability, is why the 2026 incident corpus keeps growing through pipelines with passing CI.
The industry's verification cost has not fallen at the rate its generation cost has. Nate's wedge names the reason one task at a time: spend only converts to results where checking is cheap. The 2026 corpus is what accumulates in the layers where it is not, one green build at a time, all as designed.
Motivating piece and the Google/MIT scaling study.