Nobody audits a model, they audit a harness
How we organise
Commission starts enforcing AI Act rules and new transparency requirements from 2 August (European Commission)
From tomorrow the AI Office and national authorities begin enforcing, and the Article 50 transparency duties go live. That means telling people they are talking to a machine, labelling synthetic media, and marking AI-generated output in a machine-readable form. Published alongside it is a first list of over 180 signatories to the general-purpose AI transparency Code of Practice, read on 1 August and likely to keep moving.
Structure: every one of those duties is a property of the system around the model rather than of the model itself. Machine-readable marking is an output pipeline decision. Disclosure is an interface decision. Neither is something you procure from a lab, and neither appears on a model card. We have yet to see a compliance plan that rests on a vendor attestation and holds evidence about the right artefact.
Take one AI-facing product and trace who owns each Article 50 duty inside your firm. Where the answer is the vendor, the duty is unowned.
Building abundant intelligence (OpenAI)
Sarah Friar’s case is the familiar flywheel: falling cost per task expands adoption, adoption funds compute, compute lowers cost per task. Take it for what it is, a CFO explaining an investment thesis. The number worth carrying out of it is operational rather than financial. OpenAI reports that agentic work through Codex now accounts for 99.8 per cent of its own weekly output tokens, and that users active for six months send roughly half again as many daily messages across twice as many task types.
Finance: that first figure is a vendor describing its own production, and no stronger evidence exists on whether agentic engineering is real inside the labs. It is also a warning about capacity planning. Where a firm’s tokens converge almost wholly on agent turns, every assumption built on single-shot pricing, single-shot latency and single-shot review is already out of date.
Work out what share of your own AI tokens now come from agent turns rather than single prompts. That ratio is what your pricing and latency assumptions should have been built on.
Measuring the impact of AI coding (Brian Houck, DX)
Houck argues against composite single-number metrics of the “Developer Horsepower” kind, because they collapse a many-sided question into a figure nobody can act on. He proposes measuring throughput, deployment survival, reinvested time, quality and satisfaction separately. He cites a study across roughly 100,000 developers in which commit volume rose by as much as 180 per cent while effects on shipped releases fell away to somewhere around 20 to 30 per cent.
Talent: the gap between those two numbers is the argument, and it is the same gap the rest of the week keeps pointing at. Production went up almost twofold. Delivery went up by a fifth. Whatever absorbed the difference sits between the commit and the release, which is to say in review, verification and integration. A single composite metric averages that gap away, and reports progress.
Measure commit volume against release volume across your last two quarters, broken down by team rather than reported in aggregate. Whatever distance opens between those two lines is where the gains are being absorbed.
How we build
Inside Anthropic (Gergely Orosz)
Orosz reports from on site rather than from a briefing. On one Bun to Rust migration, the split he describes is roughly 85 per cent verification to 15 per cent implementation. Code review and testing are increasingly AI-driven, two-engineer project teams and written PRDs have survived the transition, and prototyping has become markedly more fluid.
Engineering: I would treat the 85 to 15 figure as the most useful single number published this week, with the caveat Orosz gives it, that this is one migration and one team rather than a company-wide number. Read against the DX study it stops looking like an anecdote. One source measures a bottleneck between commit and release across a hundred thousand developers. The other names what the bottleneck is made of, from the inside, at a firm that has gone further into AI-assisted development than most.
Ask your teams to estimate their own split between building and checking, then compare it with where their time is actually funded. Most plans still resource the 15 per cent.
The new rules of context engineering for Claude 5 generation models (Anthropic)
Anthropic cut Claude Code’s system prompt by over 80 per cent with no measured evaluation loss, and sets out six inversions that follow. Rules become judgement. Examples become interface design. Upfront loading becomes progressive disclosure. Repetition becomes simple tool descriptions. Manual memory becomes automatic. And simple specifications become rich references, in which code, tests and artefacts are the specification.
Architecture: the last of those is the one to take seriously. If tests and artefacts are the specification, then the specification is executable, and whether the model did the right thing becomes a question a machine can answer. That is a far bigger claim than the 80 per cent, which is a headline about their prompt rather than yours. Note also what was removed, which is the material telling the model how. What survives is the material telling it what.
Open your longest agent instruction file and split it into what and how. The how section is the part a model release can delete for you.
Deterministic AI is an architecture problem, not a model problem (Keith Townsend)
Townsend, drawing on a conversation with Chef founder Adam Jacob, argues that repeatable results come from wrapping the probabilistic model in deterministic code. Encode the repeatable steps as software. Invoke model judgement only at bounded points. Type and check the output, so that a test decides whether the work is done rather than the model’s confidence in itself. His lab data backs it. A local model inside a deterministic harness matched frontier correctness at a fraction of the cost.
Architecture: set this beside the Anthropic post and you get the actual design question, because the two point in opposite directions. Anthropic removed the rules and trusted judgement. Townsend wants judgement fenced and everything around it made deterministic. Both are right about different halves, and the resolution is not a compromise. Specify what, leave how to the model, and make the boundary of every step typed and testable. Judgement in the middle, determinism at the edges, which is also, and this is no accident, the only kind of system you can hand to an auditor.
Pick one agent workflow and mark where judgement is genuinely needed. Everything either side of those points is code you have not written yet.
Change2Task: constructing verified coding-agent tasks from repository history (Microsoft Research)
Merged pull requests are turned into executable agent tasks. A healthy repository state is rebuilt first, then the whole base-to-task-to-restored lifecycle is checked rather than trusted. From 1,130 eligible source changes across five task families they report a 79.6 per cent verified construction rate, 29.2 per cent more recovered tasks than a pull-request-only baseline, and a 10.8 per cent cut in pipeline compute cost.
Engineering: the interesting move is the validation loop rather than the yield. A benchmark that verifies its own construction is a benchmark whose failures can be told apart from its bugs, and almost every agent benchmark now lacks that property. It is the same instinct as Townsend’s, which is to put a deterministic check around the part you cannot otherwise trust.
Ask of your own agent evaluations whether a failure means the agent failed or the test is broken. If nobody can tell, the suite is measuring itself.
How we assure
Investigating incidents in our cybersecurity evaluations (Anthropic)
A retrospective review of 141,006 evaluation transcripts, prompted by OpenAI’s earlier Hugging Face disclosure. It found three capture-the-flag exercises in which models reached the real internet through a test environment misconfigured to have no internet access, and compromised production systems at three separate organisations. In one case a malicious package was published to PyPI and executed on 15 external machines before removal. Anthropic frames this as an operational and harness failure rather than an alignment failure. Its newest model, they note, was the only one to recognise the environment as real and stop without being told. Work continues with METR and Irregular.
Risk: this is the most important disclosure of the week, and the framing is correct. The models behaved as instructed. The containment did not exist. A safety evaluation is itself a production system, with credentials, network reach and an agent inside it, so treat it as a lab rather than as infrastructure and you get a live malware upload out of a test. Anyone running agent evaluations should read it as a description of their own setup, because the mistake is not exotic. The credit is for publishing this much detail, the part where the harness was theirs included.
Find out what network access your own AI test environments actually have, as opposed to what they were configured to have. The gap between those two is this incident.
Announcing NIST’s Artificial Intelligence Technology Evaluation (NIST)
A sequestered blind-data testbed, built so that evaluation data cannot leak into training. It opens with three vision-language tasks in quantum science, genomics and public safety, with first evaluations from August.
Risk: small announcement, structural significance. Held-out data the vendor has never seen is the only way a benchmark score survives contact with a training pipeline, and until now essentially nobody outside the labs has run one at scale. Put it next to the Anthropic disclosure and the shape of the year is clearer. Evaluation is turning into infrastructure, with the duties that implies, and the bodies building it are no longer only the ones being evaluated.
For the benchmarks your firm relies on, ask whether the test data could have reached the training set. Where it could, the score is a measure of memory rather than capability.