The model you bought is not the model you got
How we organise
GPT-5.6 Sol xhigh uses twice the tokens of GPT-5.5 (Vincent Schmalbach)
Schmalbach compared two fourteen-day windows of his own Codex logs, 1,667 sessions against 1,715. Tokens per session rose 2.25 times. Per-token pricing did not move, and a charge for cache writes appeared. His conclusion is blunt. A fixed quota now buys less than half the work it bought before, with no price rise anywhere on the invoice.
Finance: the invoice line did not change, so nothing triggers a review. The consumption did. Any budget anchored to a per-token rate or a seat count will drift without a single approval crossing anyone’s desk. We have watched teams renew on last year’s usage assumptions and find the gap at quarter end. One developer’s logs are not a controlled study. The direction is the part a finance function should care about.
Pull your own token consumption per session for the last two months and plot it against the model version you were running. If that line rises while the invoice holds flat, your quota is repricing itself.
White House to host AI companies to review a voluntary testing framework (CNBC)
The framework was ordered by Trump in June. Developers would give government up to thirty days of early access to covered frontier models for cyber-capability assessment. The benchmark used in that assessment is classified. The order bars anyone from turning the scheme into mandatory licensing or preclearance.
Structure: a pre-release testing window is becoming the shape of AI governance, and it is arriving as voluntary practice before it arrives as law. Firms that already run a defined gate before a model reaches production will find the paperwork trivial. Firms that deploy on vendor assurance will be asked for evidence they do not hold. The gap between those two positions is organisational, not technical.
Write down who in your organisation may say no to a model going live, and what evidence that person needs before saying it. If the person does not exist, you do not have a gate.
How we build
The Shape of Things to Come, Part 1 (Steve Yegge)
Yegge argues that reusable agent harnesses are a dead end, and that a harness has to be chemically bonded into the application it serves. His own Gas Town harness fell apart when Opus 4.7 developed a habit of never converging on the work. Within hours AWS published the opposite bet: Kiro’s three separately built harnesses, one in TypeScript for the IDE, one in Rust for the CLI and one in Python for the web, are now a single server-side harness behind a defined protocol.
Architecture: both sides argue from experience and both are right about something. A tuned harness beats a generic one. Three harnesses drift apart and cost three times as much to keep alive. The decision is not which vendor to pick. It is where you draw the line between the part that must fit your particular work and the part that should be shared across every client and every team you run. Draw that line too high and you maintain three of everything. Draw it too low and your agents run inside a scaffold built for somebody else’s problem. Few organisations seem to have drawn it at all, which is probably why this reads as a matter of taste rather than a design decision carrying a bill.
Name that boundary in your own agent stack this week, and write it down somewhere the whole team can see it.
How we assure
Endpoint Accuracy Index v1.0 (Artificial Analysis)
The index scores how far a given API provider’s accuracy falls below a self-hosted reference deployment of the same model. Three evaluations carry equal weight: tool calling, hard reasoning and long-context recall. Every result carries a confidence interval and a flag for whether the endpoint sits statistically below reference. The scores are point-in-time snapshots rather than live monitoring.
Risk: the model name on a contract does not fix the accuracy you receive. Quantisation, serving configuration and routing all sit between the weights and your answer, and none of them appear in a procurement document. We have seen procurement treat the model name as the specification. A team that measured a model in one place and deployed it in another is likely carrying an assumption nobody tested. There is now a published method for testing it.
Take the evaluation you trust most and run it against your production endpoint rather than the vendor’s playground. Any gap between the two is the number to put in front of your supplier.
MirrorCode (Epoch AI and METR)
MirrorCode asks a model to reimplement a whole program from scratch and match the original’s output exactly on held-out end-to-end tests. Twenty-five target programs, across several domains. The economics are the finding rather than the scores. One task cost 2,600 dollars across nineteen unattended days. Claude Opus 4.7 rebuilt a sixteen-thousand-line Go bioinformatics toolkit in fourteen hours for 251 dollars.
Risk: those two numbers are the same benchmark, and the spread between them is the thing to plan around. Long-horizon autonomous work has no reliable unit cost yet, so a fixed-price commitment against it is a bet on which end of that range you land. The fourteen-hour result is what a vendor demonstration shows you. The nineteen-day result is what a procurement contract has to survive.
Before you commit to a fixed price on any autonomous delivery, run the same task three times and record the cost of each. Quote against the worst one.