The meter cut the price, and the checking was the work
Metering finished coding-agent sessions cut the cost of a solved task by up to a third, Trail of Bits counted the regressions in its own agent patches, and AI found PHP's decade-old flaws days before attackers used them.
How we Organise
Microsoft Research and UC Berkeley built a plugin that wraps the main coding agents unchanged, reads each finished session in a repository, and changes what the next one pays for. Facts the sessions keep rediscovering are stated up front, tools that go unused are pruned, and a turn stuck in a failing loop gets one interruption. On SWE-Bench the mean cost of a solved task fell 18.4 to 30.9 per cent across five agent and model combinations. It fell in all thirty cells.
The quantity measured is the one our finance practice asks for: spend divided by successful outcomes. This is the first reading we have logged that moves that number down. It comes from the method’s own authors on a public benchmark, so the share a client should expect on its own code is an open question. A strain against this position has stood since 5 September, and today’s source does not settle it.
Consumption per unit of output is falling, or the reason it is not is recorded.
How we Build
One in six agent-written patches that maintainers merged broke something else
Trail of Bits reviewed every patch its engineers wrote with agents and had merged into open-source projects over two months of work. It confirmed 21 regressions across more than 130 pull requests. None broke a security protection. Its answer is a checker that makes the agent reproduce each suspected regression on a live deployment before an engineer sees it. That produced 25 per cent fewer false positives than a baseline agent told to review the same work.
The practice we ask of clients is that a reviewer’s correction becomes a case the suite still runs. That is exactly the loop on show. The 21 confirmed regressions are the set the checker was evaluated against, and the tasks an earlier version failed become the next version’s benchmark. The firm accepted a trade with eyes open: fewer false alarms at the price of some real regressions left borderline.
The case a reviewer’s correction became is a case the suite still runs.
How we Assure
Wiz Research and Calif, working with OpenAI, used frontier models to find and prove unauthenticated remote code execution in PHP’s core engine. The two vulnerabilities scored 9.2, and PHP runs about 70 per cent of websites. The PHP project shipped fixes on 21 October after a leak compressed the timeline. AWS analysts observed exploitation beginning on 17 October.
We hold that you attack your own agents before somebody else does. This is that principle at internet scale, from both directions at once. AI is how the defenders got there first, and the somebody else arrived days later rather than months. The position’s premise about tempo holds, and the margin it buys is compressing. A strain raised against this principle on 20 September stands open, and this case does not answer it.
Attack your own agents before somebody else does.