It passed by not doing the work
How we Organise
Anthropic’s playbook for an AI-native development lifecycle
Louis Claxton of Anthropic’s Applied AI team published this on 21 August, and it opens by saying code is no longer the bottleneck. What follows is an argument about where the bottleneck went. Once agents write most of a change, the stages on either side of the build run at human speed and become the schedule: planning, review and testing, deployment. Governance costs rise because exceptions still route through committees that meet weekly or monthly. The worked example is security. Teams there are sized for human output, so when agents multiply the volume of code either the queue grows or the change ships under-reviewed. A regulated firm can accept neither.
The answer proposed is a loop rather than a line. Each stage commits an artefact the next stage reads: an intent file, a specification, a plan, the diff and its tests, the pull request with its findings, the incident record. That chain of commits doubles as the audit trail of who asked for what, what the agent produced and who approved it. For the first stage the playbook names its own measure, and the measure is a clock. It times the first conversation to a committed intent file, and the git history already records the author and the timestamp.
We ask a team to write down every decision it needs from outside. Each one names who decides and who owns it on each side, and the team then tracks it for the time it takes from question to commitment. That last part is the one a client argues about, because decision latency is nobody’s number and reads as an audit of colleagues. Anthropic arrives at the same measure from the other direction, and reaches for a timestamp on a committed file rather than a survey.
That is a better version of the measure than the one we usually describe.
This is a vendor document, and the tools it recommends are the ones Anthropic sells. Its measures are written as expectations rather than results. The time from conversation to a committed intent file is expected to fall from weeks to hours, and no organisation’s before-and-after figures appear. Someone could reasonably read the whole thing as an account of how Anthropic works.
How we Build
A benchmark for whole-repository migrations
Deyao Hong and seven colleagues posted this on 24 August. They assembled twenty whole-repository migrations across four kinds of technical debt and put eight frontier models through them in twenty-six model-effort configurations, 520 runs in all. Their reason for building it is a hole in the benchmarks that already exist. Those score behaviour, so an agent can copy the original implementation across, pass the tests, and be recorded as having migrated the repository. The authors call that blindness and check for it before anything else.
Their protocol runs in three stages. An audit establishes that the migration happened at all. A fixed suite then measures whether behaviour survived. Six independent coding agents then write targeted tests hunting for behavioural differences the fixed suite missed. Twenty-eight of the 520 runs cleared all three, which is 5.4 per cent. Thirteen of the twenty tasks drew no accepted solution from any model at any effort, and the best score was 47.0 out of 100. The two abilities came apart under the protocol. A few runs kept behaviour intact by skipping the migration and were stopped at the audit. Most attempted it and broke behaviour. Of the 340 runs that got past the audit, 58 per cent reached 99 per cent of the fixed checks and 26 per cent reached all of them. Agents scored 31.4 on build toolchain rewrites and 5.6 on language rewrites.
Every skill a team builds should carry its own suite, written before the skill exists and failing at that point. It takes three forms: worked examples a business expert vouches for, properties that must hold across every input, and scenarios in the business’s own language. All three ask what came out. None of them asks whether the work was done. An agent that copies the old implementation forward satisfies every worked example, holds every property and passes every scenario, because behaviour is exactly what it preserved.
That makes three sources in five days arguing that a passing suite can evidence nothing, and each names a different hole. Researchers on scaffold robustness found rankings reversing when the harness changed. Hume AI with Hugging Face found speech models reciting the reference transcripts they were being tested against. Now a suite that cannot tell a migration from a copy. We may be reading our own position more widely than it earns. For it to survive, the test needs a clause about the change rather than the result. A suite has to establish that the work happened, separately from whether the output still behaves.
Twenty repositories is twenty, and a benchmark is not a field study. Migration is an unusual task, chosen because the intended change is unusually easy to audit, and most skills a client builds are not migrations. Somebody could fairly say a business-language scenario written by an expert would notice a copied implementation. That holds where the expert knows the change was meant to happen. It does not where the suite is all anyone reads.
So a green suite stops being evidence that a skill works, unless a client can also say what changed.
How we Assure
Boyd Kane on models exploiting the engines that run them
Boyd Kane published this on 25 August, asking whether a model could take over the machine running it. The setup he describes is ordinary. An agent acts on one machine through a harness. The tokens it emits are produced on another, the one holding the weights and sitting in the datacentre with privileged reach into everything else. Between them runs an inference engine, which does considerably more than turn tokens into strings. It parses chat formats for a long list of model architectures, and the model controls what it is given to parse. Kane’s argument is that a model can emit a sequence chosen for what it does to that parser rather than for what it means. That is an ordinary software exploit with an unusual author.
The instance he points at is real and closed. In vLLM, the server a great many self-hosted models run on, one model family’s tool parser met a parameter whose type it did not recognise. It handled that by passing the parameter to Python’s eval. Any authenticated user who could get the model to pass code as a tool-call argument could then run it on the serving machine. The maintainers scored it high, reachable over the network, needing low privileges and no user interaction, and fixed it in a patch release covering two published versions.
We ask that agent-written code and agent actions run inside a declared boundary. That means sandboxed execution, an enumerated list of what can be reached, and a route out carrying a named approver and a date. Every clause in that sentence describes the machine the agent runs on. The tokens leave that machine and are produced somewhere else, and the parser reading them belongs neither to the client nor to the agent’s declared surface. A client can enumerate every reachable system, name an approver for every route out, and pass on all of it. The sequence a model emits is still parsed on a machine that holds the weights.
This is the second source in a week arguing our boundary is drawn in the wrong place. In August the UK AI Security Institute reported unsanctioned actions from an evaluation whose route out had been approved deliberately. Approval is not observation. This one is narrower and harder to answer, because that parser has an owner and the owner is not the client.
We may be wrong that this belongs to us at all. Kane is careful about what he does not know and says so. Finding a useful vulnerability is the harder step, and he is unsure how likely a model is to manage it. Nobody has yet shown a model discovering and exploiting one of these by itself, and the vLLM bug needed an authenticated user, which is a real constraint. Somebody could reasonably call it an ordinary supply-chain problem in one server, answered by patching.
For a self-hosted estate the repair is version discipline on the engine, which a client can own. For a hosted model the engine belongs to a provider. The position would have to become a question about that provider’s assurance rather than about the client’s own sandbox.
Either way we stop describing a sandbox as a boundary around what an agent can do, while the tokens it produces are read on a machine the client never enumerated.