It ran, and it could not keep up
How we Organise
The smallest model in the comparison was the most expensive to run
Songyuan Li, Ahmed Abdelmoniem and Shiqiang Wang posted this on 26 August and revised it on 30 August. Their subject is routing: which model handles each step of a multi-agent workflow, decided step by step rather than once at the start. They measured it on four public benchmarks. Cost is reported as energy in joules, against a budget set for each benchmark.
The baselines carry the finding. On MBPP the budget was 4,500 joules. Qwen 3.5 2B used 14,308 joules and passed 64.5 per cent. Qwen 3.5 35B used 4,804 joules and passed 93.9. Qwen 2.5-Coder 0.5B used 5,571 joules and passed 8.0 per cent. Every fixed single-model baseline went over budget on both coding benchmarks. Their router finished at 3,376 joules and 79.4 per cent, and was the only method inside the budget there.
We tell a client that a cheaper model which is wrong more often costs more than the expensive one that is right. That has been an argument from first principles, and a client has always been free to disagree with it. Here it is a measurement, and the direction is not marginal. The smallest model in the comparison was the most expensive to run, by a factor of three over the largest. Failure is paid for in repeated work rather than in the price of a call.
Energy on four benchmarks is not what a client pays, and the ranking could differ where pricing does not follow compute. This is also a router paper reporting its own baselines, and baselines are chosen to be beaten. The direction survives both of those, because the gap between 14,308 joules and 4,804 is not the sort of thing a favourable comparison manufactures.
How we Build
Forty-eight coding agents wrote one specification, and the failures arrived together
Javier Ron, Benoit Baudry and Martin Monperrus, from KTH and the University of Montreal, submitted this on 18 June. They rebuilt the 1986 Knight and Leveson experiment with coding agents in place of students. The specification is the original Launch Interceptor Program. Forty-eight implementations were admitted, drawn from 69 combinations of harness, model and language, covering five agent systems, 23 models and three languages. All 48 then ran a shared campaign of 1,000,000 randomised inputs against one oracle. A version fails a case if any of its 241 output bits differs.
Independence predicts 115.36 cases where versions fail together. They observed 429, at a z-score of 29.20. That is one half of it. The other half is that redundancy paid: voting over all 17,296 triples dropped the mean failure count from 387.44 to 130.99, and 11,844 of those triples produced no failures at all, against 27 of the 48 single versions. Both distributions have a median of zero. The movement is in the tail. At the 99th percentile a single version fails 6,004 times and a triple fails 419.
We hold that the construction should be chosen deliberately, between deterministic code, a single model call, an agent and an orchestration, with the simpler option tried or ruled out on the record. Our test is whether that choice was made and recorded. A team can pass that test completely and still be wrong about what it bought. Recording “we ruled out a single call because we need independent review” satisfies the practice. Independence is the thing these numbers say does not arrive. What does arrive is a smaller worst case, which is worth having and is a different product. For the practice to survive it has to ask what the chosen construction delivers, not only that the ladder was climbed on the record.
This is not the first source to argue it.
The fine-tuned single model call that beat every text-to-SQL scaffold, reported here on 30 August, made the same point from the other end: the rungs do not name what actually decides the result.
We may be wrong, and the authors themselves read the data the other way. Their stated conclusion is that this is the strongest evidence to date that N-version programming with coding agents is a useful engineering strategy. Both readings fit the same table. They name their confounders too, a shared specification and overlapping training corpora, and the work is a preprint. Code generation with a bit-exact oracle is the easiest case there is, and nobody has run this on agents reviewing work rather than producing it.
If it stands, we stop telling a client that moving up to an orchestration is itself evidence of a more reliable construction. Every client running three agents as independent review is holding a smaller worst case and calling it more than that.
How we Assure
Anthropic froze its training environments for a month to catch up with itself
Anthropic published this on 31 August. It describes an automated review of every reinforcement learning environment, before and during training runs, built up over several model generations. By spring 2026 it was, in its own words, “producing RL environments at a more rapid pace than ever, faster than our systems could vet them”. Flagged environments waited on a person to adjudicate, while “reward hacks and misconfigurations started outpacing our ability to filter or fix them”. In April it froze all changes to production RL environments for roughly a month and rebuilt the stack. During that freeze it flagged over 10 per cent of the environments in its production mix for reward hacking, broken tasks and misconfiguration. Each was reinstated only once fixed. Roughly 150 product engineers were moved onto security, reliability and privacy.
We ask that nothing capable of changing an agent’s behaviour changes without a recorded review, and we name what that covers: the model, the prompt, the tools, the corpus and the identities arriving with the supply chain. The check is whether the review exists and is recorded. All of that was true here and it was not enough: the review existed, ran automatically, covered the whole population and produced the flags. What it could not do was keep up, so defects reached training runs while the practice read green throughout.
The binding constraint was the rate at which review could be finished, and our practice does not mention rate. No client would think to report it. A queue of unadjudicated flags looks like a backlog rather than a finding. For the practice to survive it has to ask whether review is keeping pace with the rate of change.
We may be wrong about how far this reaches. One firm is describing itself here, the figures are the ones it chose to publish, and an independent review with METR is planned rather than delivered. A frontier lab generating training environments is not a bank changing a prompt, and a reasonable person could hold that ordinary enterprise work leaves far more headroom.
If it stands, we stop telling a client that a recorded review over every change is evidence the change is being reviewed. The question we ask a client becomes how long the queue is, and how fast it is growing.