The price fell and the bill went up
How we Organise
Vercel publishes a monthly index from traffic through its own AI gateway. The August edition covers July. Token volume grew 59 per cent. The bill was 37 per cent larger. The average price per token fell 13.6 per cent. Hold June’s mix of models constant and that average would have stayed flat, so the fall came from what companies routed rather than from anything getting cheaper. Thousands of teams ran more than 10 million tokens in both months. Three in four changed at least a tenth of their model mix, and three in five changed at least a quarter. The median team’s price per token fell 2.9 per cent, and only one team in six stayed within 5 per cent of where it started. A quarter cut theirs by more than 30 per cent. Another quarter paid at least 20 per cent more. Eighty-one per cent of July’s tokens ran on models that were not on the gateway six months earlier.
We tell a client to work out what one successful outcome costs. The price per token is one input to that and never the answer. These numbers are that argument at market scale. Over four weeks, on the same traffic, the price per token and the bill moved in opposite directions, and a client watching only the first would report that AI got cheaper in a month when it paid more.
Vercel sells the gateway, values spend at list prices rather than at negotiated bills, and sees one slice of demand. Moving work to a cheaper model is a rational answer to price, and someone could fairly say nobody is being misled here. That holds for any single team. It leaves the aggregate reading wrong, and the aggregate is what a board gets shown.
How we Build
Measuring benchmark optimisation in speech recognition
Researchers at Hume AI published this with Hugging Face on 21 August. They ran three probes across eleven widely used open-source speech models. The first flags clips where an ensemble of models with low phoneme error unanimously disagrees with the benchmark’s reference transcript, then checks a sample of those against human annotation. They found probable reference errors in 40 per cent of the VoxPopuli clips analysed, about 3 per cent of all reference words. Several of the top-scoring models reproduced those errors between 18 and 30 per cent of the time, and the models with the lowest reported error rate did it most. The second probe silences a number in the audio and asks for a transcript. Models supplied the number anyway, one of them a year. The third measures whether a model switches spelling to match whichever corpus it is being tested on. Random choice scores 50 per cent, and several models reach roughly 90. Then the researchers gathered fresh recordings from the same sources, made after every model’s training cutoff, and the behaviour weakened or disappeared.
We ask that every skill carry its own suite, written before the skill exists, in worked examples a business expert vouches for, properties that must hold across every input, and scenarios in the business’s own language. The suite is how a team knows whether the work is good. This is the second source in three days to argue that passing one can evidence nothing. Researchers looking at scaffold robustness on 22 August found rankings reversing when the harness changed, and this week’s finding is models reciting the benchmark they were meant to be tested against.
Our test asks whether a suite exists and takes those three forms. It does not ask where the examples came from, or whether the model has already read them. A team that assembles its examples from a public corpus passes the test as written, and has measured how well the model remembers that corpus. We may be wrong about how far this reaches. For the statement to survive, though, the test needs a clause it can read: the examples must be ones the model could not have read.
Speech recognition is not reasoning and it is not code, and the authors publish leaderboards that compete with the ones they criticise. A suite drawn from a client’s own live cases was never published. It was never at risk. That is right, and it is not where most teams start, because the first suite is usually the public one.
So a green suite stops being evidence that a skill works, unless a client can say what in it the model could not have read. Every client running one has a passing number and no account of what it measured.
How we Assure
Anthropic on widening access to its strongest cyber model
Anthropic published this on 21 August. Claude Security scans can now run on Claude Mythos 5 for customers on Enterprise plans, returning vulnerabilities and suggested patches rather than model output. The same model is going into partners’ existing security products, where an end user receives only the artefact that product exists to produce. A 35 million dollar fund will give credits to organisations patching open-source projects. A verification programme that already reduces safeguards for vetted defenders on two other model families will widen, with access to this one to follow. The stated reason is that direct access to a model is the riskiest arrangement, and that returning a patch or an alert instead lowers the risk.
Everything able to change an agent’s behaviour should pass a change review, and we name the surfaces: models, prompts, tools, retrieval corpora, the classifiers beside them, and the machine identities a supply chain brings. Where a client consumes a model rather than building one, we ask that the provider’s assurance be recorded rather than assumed. All of that can be done here, and Anthropic has published the assurance to record.
What our list has no line for is the patch. Every named surface sits inside somebody else’s product, and the thing crossing into the client’s estate is a suggested fix for their own code. A client can hold the review closed on all six surfaces and still merge machine-written code into a system they answer for. We may have drawn that boundary in the wrong place. The test needs the artefact a model produces for you added to the things reviewed before they land.
Anthropic is describing its own safeguards, and a customer cannot test the claim that a user is unable to steer the model toward an exploit. A suggested patch might be code like any other, already meeting a client’s review standard. That is fair wherever the standard reads a security patch as closely as it reads a feature. Not every standard does.
So we stop describing change review as covering everything that can alter behaviour, while a model nobody in the client’s building may inspect writes into their repositories.