Nobody audits a model, they audit a harness
Transparency duties that live in the system around the model, a verification-to-implementation split of 85 to 15, and an evaluation harness that reached the open internet.
How we organise
The Commission starts enforcing the AI Act
From 2 August the AI Office and national authorities begin enforcing, and the Article 50 transparency duties go live: tell people they are talking to a machine, label synthetic media, mark generated output so a machine can read it.
We tell a client that every one of those duties is a property of the system around the model rather than of the model itself. Machine-readable marking is an output-pipeline decision. Disclosure is an interface decision. Neither one is procured from a lab, and neither appears on a model card. We have yet to see a compliance plan that rests on a vendor attestation and still holds evidence about the right artefact. Where the owner turns out to be the vendor, the duty is unowned.
Building abundant intelligence
Agentic work through one vendor’s coding tool now accounts for 99.8 per cent of its own weekly output tokens.
Our reading is that this is a vendor describing its own production, and that no stronger evidence exists on whether agentic engineering is real inside the labs. Where a firm’s tokens converge on agent turns, every assumption built on single-shot pricing, latency and review is already out of date.
Measuring the impact of AI coding
Across roughly 100,000 developers, commit volume rose by as much as 180 per cent. The effect on shipped releases fell away to 20 or 30.
We judge the gap rather than either number. Production nearly doubled. Delivery rose by a fifth. Whatever absorbed the difference sits between the commit and the release, in review, verification and integration.
How we build
On one migration the split was roughly 85 per cent verification to 15 per cent implementation. It was one team and one migration.
We hold that most plans still resource the 15 per cent. Read against the developer study above, this stops looking like an anecdote. One source measures a bottleneck between commit and release across a hundred thousand developers, and this one names, from the inside, what the bottleneck is made of.
New rules of context engineering
A system prompt cut by more than 80 per cent with no measured evaluation loss, and six inversions that follow. Rules become judgement. Upfront loading becomes progressive disclosure. Specifications become code, tests and artefacts.
Our position is that the last of those is the one to take seriously, and not the 80 per cent. If tests and artefacts are the specification, the specification is executable, and whether the model did the right thing becomes a question a machine can answer. What was removed was the material telling the model how.
Deterministic AI is an architecture problem
Repeatable results come from wrapping a probabilistic model in deterministic code. Encode the repeatable steps, invoke judgement only at bounded points, then type and check the output. One local model inside such a harness matched frontier correctness at a fraction of what the frontier model cost to run.
We read this beside the item above and find them pointing in opposite directions. One removed the rules and trusted judgement. This one wants judgement fenced and everything around it made deterministic. Both are right about different halves. Judgement sits in the middle, and determinism at the edges.
Building verified agent tasks from repository history
Merged pull requests become executable agent tasks, with the whole lifecycle checked rather than trusted.
We want a suite that can tell a failing agent from a broken test, because a suite that cannot is measuring itself. A benchmark that verifies its own construction has that property, and almost none do.
How we assure
Investigating incidents in cybersecurity evaluations
A review of 141,006 evaluation transcripts found three exercises in which models reached the real internet through a test environment misconfigured to have none, and compromised production systems at three organisations. A malicious package was published and ran on fifteen external machines before removal.
We treat a safety evaluation as a production system in its own right, with credentials, network reach and an agent inside it. The models behaved as instructed. The containment did not exist. The framing offered, harness failure rather than alignment failure, is the right one.
NIST’s blind-data evaluation testbed
A sequestered testbed, built so that the evaluation data cannot leak into a training set. It opens with three vision-language tasks.
We ask whether the data behind a benchmark could have reached the training set. Where it could, the score measures memory rather than capability. Evaluation is turning into infrastructure, and the bodies building it are no longer only the ones being evaluated.