The number was right and the thing it stood for was not
How we Organise
The spend moved twenty-eight times and the budget was set once
Justin Reock published DX’s Q2 2026 report on 22 July, drawing on more than 500 organisations. AI wrote 34 per cent of merged code in the first quarter and 52 per cent in the second. Adoption across the industry passed 90 per cent. Developers save four to six hours a week.
The spend moved faster than any of that. Median quarterly AI spend per organisation went from roughly $1,500 to roughly $44,000, and in the tech sector it rose nearly twenty-eight times. Over the same four quarters the share of developer time going into new features rather than maintenance did not move at all. Reock’s own conclusion is that the hours saved are not converting into anything.
Spend that moves twenty-eight times inside a year is not a variance against a budget, and much of what we argue to a client about model spend rests on saying so. This is the first time we have seen it measured across a population. An annual budget is not wrong here by a margin somebody could tighten next year. It is the wrong instrument.
We could be reading it too widely. The population is one vendor’s customers, and customers who had already bought a tool to measure their developers. Spend starting at $1,500 is a small base, so the multiple flatters. Reock does not publish the sample behind each figure. Somebody could fairly say a young category always moves like this and then settles, which would make an annual budget wrong for two years rather than wrong in kind.
If that flat share holds, hours returned to a team stop counting as a benefit on their own. They become one when something is built with them, and four quarters of this cannot yet show that happening.
How we Build
Every leading system found the right code and put the wrong one first
Aaryan Kapoor and Md Abdullah Al Hafiz Khan submitted ExecRetrieval on 1 September. Embedding-based retrieval sits underneath coding agents. Benchmarks usually ask whether it returns code on the same topic. These authors asked whether it can separate code that works from code that nearly works.
They built a set holding the near misses. The set runs to 939 Python tasks. Each carries one canonical implementation and up to four buggy variants, every one execution-verified, each a single targeted mutation. Against that set they ran 23 dense embedding configurations and BM25 through each provider’s own interface, with paired McNemar tests and query-level bootstrap intervals. The top hosted system returns the canonical inside ten results every time. It returns it first only a third of the time. When the four leading systems miss at rank one, what sits there instead is one of that query’s own buggy variants between 91.5 and 99.4 per cent of the time. The canonical scores below at least one of its own variants in 67 to 78 per cent of queries.
Sorting a bad answer into one of two causes, the material or the reasoning, is a step we ask for and would defend. It depends on a set that can tell a right answer from a nearly-right one. Build the set the ordinary way, one canonical among code about other subjects, and every leading system passes. Put four near-identical variants beside that canonical and the same system leads with a wrong one two times in three. The score holds. Sorting then clears the material, on the word of a set that never had to make the distinction.
We may have this wrong. The variants were made mechanically rather than collected from real defects, and a single-edit variant is a hard case by construction. Returning the canonical inside ten results every time says the material was in reach, so a team reranking against an execution oracle gets it back. A fair answer is that our practice already implies the set should hold the near misses, which puts the fault in how teams build sets rather than in what we ask of them.
What we would stop saying: that a retrieval score taken on its own shows the material was there. A score shows that only when the set holds code that is nearly right.
How we Assure
The revocation completed and the earlier work went through anyway
Igor Santos-Grueiro submitted this on 2 September. He names a property, effect closure. Authority is closed once nothing issued under it can still reach an effect the application rejects, and no more can be issued. His checker reduces the question to finite control with hidden state. The checker returns a strategy, a certificate that closure is impossible, or no verdict where the evidence will not decide. The reduction and the checker both carry machine-checked proofs.
He tested GitHub, Kubernetes, NATS and Kafka. Closure failed on all four, in three ways. The interface offers no control that would close the authority. Clean visible state hides work still running. Or the model stops short of the last point at which the effect could be prevented. The GitHub tool cannot bind a merge to the commit that was reviewed, and a controlled run confirms it may merge a different one. NATS reports nothing stored and nothing pending while work already issued goes on publishing downstream. In Kafka every broker had applied the revocation, and a request authorised before it could still append.
Authority that ends with the task is one of the plainer things we say about agents, and on four widely used providers it is false. Not overstated: false, with the failure located. A team reads two things to decide it: whether the credential was short-lived, and whether the revocation completed. Both read true here while work authorised a moment earlier could still reach an effect. The claim survives in a narrower form. Authority ends once issuance has stopped and nothing already issued can still act, and that moment arrives later than the one anybody currently reads.
The scope may be narrower than we are taking it. Four providers and one author. The Kafka path studied is synchronous and nontransactional, and the checker returns no verdict where evidence is thin, so three named failures are what could be decided rather than a census. A sceptic would say this is a property of those provider contracts rather than of agent authority, and that a firm running its own services can close the frontier itself.
The sentence to retire is that a completed revocation ends what an agent may do. Every client running agents against a third-party provider has a window after revocation. Nobody has measured how long it is.