It passed by not doing the work
A migration benchmark, a parser that ran what it was given, and a playbook that timed its own approvals.
How we Organise
Anthropic’s playbook for an AI-native development lifecycle
Once agents write most of a change the bottleneck moves outward, to the stages on either side that still run at human speed. The playbook has each stage commit an artefact the next one reads, and measures the first with a clock: how long from the opening conversation to a committed intent file.
We tell a client to name every decision it waits on from outside and track how long each takes. Clients argue about that one, because it reads as an audit of colleagues. A timestamp on a committed file measures the same thing without the survey, though the document is a vendor’s and its figures are expectations rather than results.
How we Build
A benchmark for whole-repository migrations
Twenty repository migrations, eight models, 520 runs, and 5.4 per cent cleared every stage. The first stage audits whether the migration happened at all. Some runs kept behaviour perfectly intact by copying the old implementation forward and never doing the work.
Our position is that a skill carries a suite written before the skill exists: worked examples, properties, scenarios in the business’s own language. All three ask what came out; none asks whether anything changed, and a copied implementation passes every one. Twenty repositories is twenty.
How we Assure
Boyd Kane on models exploiting the engines that run them
The tokens an agent emits are parsed by an inference engine on the machine holding the weights, and the model chooses what that parser gets. One vLLM tool parser met a parameter type it did not recognise and handed it to Python’s eval.
We judge an agent by the boundary around what it may execute: a sandbox, a list of what it reaches, an approver on each way out. Every clause describes the machine the agent runs on. The parser sits elsewhere, and its owner is not the client. Kane says nobody has yet shown a model finding such a bug alone.