The limit existed and the money went anyway
How we Organise
Sixty-three times the limit was an intention and the money went anyway
Researchers have assembled sixty-three confirmed production incidents in which an AI agent ran past its budget. The incidents span 2023 to 2026 and twenty-one orchestration frameworks. Each is backed by a quoted GitHub issue. Where the reporter volunteered a figure, the dollar loss sits beside it. They sort into eight failure clusters. Agreement between raters ran at a Cohen’s kappa of 0.837 across 113 items. Forty-seven further structural entries sit alongside the catalogue. The authors then built a mitigation and measured it. It is an 1,180-line Rust library. It turns three things into compile errors: cloning a budget, spending it twice, and using it after delegating it. Each had been something an operator has to remember. On single-agent work a four-line Python counter did just as well. Both overshot zero times in thirty. The separation shows up in multi-agent delegation. The fanout race documented in eleven of the incidents overshot thirty times out of thirty under asyncio. The borrow checker refused the same pattern before it ran at all. Across five runtimes, three providers and a live test of 160 runs, they report no violations and no false refusals. Static over-reservation runs at four to six times the budget.
A limit somebody has to notice is not a limit. We say that to clients plainly. The limit has to hold with nobody in the loop. It has to have been fired at least once in earnest before anyone believes it. What stood behind that until now was argument. Sixty-three incidents is a population. In each of them the limit existed as an intention. A retry loop spent the money before an operator saw anything.
The catalogue comes from public issue trackers. It holds the overruns somebody wrote up, not a sample of everything that happened. The dollar figures appear only where a reporter mentioned one. The mitigation is one team’s library in one language. Somebody could read the whole thing as a story about immature orchestration frameworks rather than about how a firm controls its spend.
How we Build
Developers threw away more than half of what the agent told them
Four researchers mined CodeRabbit’s review comments together with what developers wrote back. The sample is 31,073 comment-and-reply pairs. They come from 10,191 pull requests across 239 GitHub repositories. Of those comments, 36.4 per cent were accepted. Another 7.3 per cent started a discussion. The remaining 56.3 per cent were rejected. Most rejections were invalid suggestions. That means false positives, repeats of a point already made, or comments about something the change did not touch. The rest misread what the developer intended, or how the project writes code. Comments about whether the code worked outnumbered comments about whether it could be maintained. Those were the likelier to be invalid. The researchers then trained lightweight models to predict which comments a developer would reject. They reached an F1 of 76 per cent.
Our advice on review is to let an agent handle everything it can handle before any person opens the diff. A finding goes to a second reader only when the developer challenges it. The routing is not what this measurement touches. What it touches is the reason we give for the routing, which is that sending the agent first spends less human attention. All 31,073 of those comments reached a developer. More than half of what they read was invalid.
A team can run that routing exactly as described. It can satisfy any check written against it. Its developers can still be working through a queue that is majority invalid. The check asks how review is routed. It never asks what share of the agent’s output survives the developer. The F1 of 76 per cent is the part that could move, because it says the invalid comments are predictable from the comment and its context. Something could therefore stand between the agent and the developer.
This is one tool on public repositories. A rejection is inferred from what a developer wrote back, not from whether the comment was right. A team that had tuned its configuration might see a different split. So might one working in a private codebase with conventions the agent had been given. Somebody could reasonably read 36.4 per cent accepted as a good return on a pass that costs a reviewer nothing. If you run this and your numbers look different, we would like to see them: transform@dromologue.ai.
How we Assure
Nothing today. The strongest candidate was a maturity model for reconstructing what an agent did during an incident. Its authors describe their own completeness figures as implementation behaviour rather than external validation. It settles nothing about anybody’s practice.