Fifty-eight per cent were never once right
How we organise
Don’t be a meat proxy (Niklas Gruhn)
When someone pastes model output into Slack instead of answering the question, reading the reply costs more than prompting the model yourself. The output is verbose, jargon-dense, and salted with plausible nonsense that the sender never checked. Gruhn extends this to code review: paste the ticket in, paste the review comments back, ship. The reviewer used Claude, the developer relayed it, and nobody in the chain built the judgement that distinguishes good output from bad. The remedy is simple: read it, understand it, check it, write the answer in your own words. Those words are the proof of competence. Without them, the person is a relay, not a contributor.
Talent: this looks like a manners problem and is a competence problem. Someone who relays output has not built the judgement that tells good output from bad, and nothing in the week forces them to. We see the cost land on the reader, which is why it never appears in the sender’s numbers. That is how the habit spreads.
Take one artefact from each team last week and ask who read it before it went out. Where nobody can answer, you have found a review step that quietly stopped happening.
Devtools must be open source (David Crawshaw)
Five years ago almost none of the engineers Crawshaw asked had written any software for themselves, and he thought that was rational. Learning a foreign codebase took weeks; returning to your own patch a year later was worse. Two things changed the arithmetic. Fetch the source, build it locally, record why you changed it. Then run a nightly job that rebases onto upstream, checks the software still works, and replaces the running version. Maintenance was the expensive half, and that half has become a cron entry. Plugin systems and configuration files were a workaround for a cost that has gone; the source code is now the extension system. The one exception that matters: Claude Code is closed, so you get the hooks it offers or nothing.
Structure: the build-or-buy case for internal tooling has moved. Few of the firms we work with have repriced it. A highly configurable task manager, CMS or CRM earned its licence when configuring it beat building it. Assembling and personalising is now a real option, even for a small team. What follows is a procurement rule. A tool that you cannot read is a tool that you cannot bend.
Take the three most heavily configured tools in your estate and price what each would cost to fork and hold under a nightly rebase. If nobody has run that number, your policy is priced on last year’s costs.
How we build
How we built a realtime system for responsive voice AI in six months (Justin Uberti and Zahan Malkani, OpenAI)
OpenAI rebuilt the infrastructure behind GPT-Live voice in six months. Almost none of the work was model work. The biggest latency gains came from rewriting the media layer and the inference orchestration: their worst-case response time dropped to where their previous median had been. Under load testing, the component that saturated first was not the GPU but a CPU-side process in the orchestration path. Shadow testing caught it; the capacity plan had not accounted for it.
Engineering: the binding constraint on an AI system is rarely the model itself, whatever the procurement conversation assumes. It is the orchestration, the media handling, the network path, the queue management. Most capacity plans are written against the GPU count, which is the part of the system that nobody had to go and discover. OpenAI spent six months discovering the rest.
Find the last time anyone load-tested your agent path end to end rather than benchmarking the model alone. The component that saturates first is rarely the one in the budget.
Is memory the moat? (Ian Ye, Wafer)
Ye ran Kimi K3 on AMD hardware and compared it to NVIDIA’s Blackwell. On raw throughput, AMD loses by about 40 per cent. On cost per token, AMD wins: 48 tokens per second per dollar against 33. The secondary coverage collapsed these into one claim (“better performance than Blackwell”) and dropped the distinction. A workload bound by latency needs the faster node. A workload bound by budget needs the cheaper one. They are different purchasing decisions, and the vendor comparison in front of most buyers reports only one of them.
Architecture: performance per dollar and absolute throughput are different claims, and procurement that conflates them buys the wrong hardware. The summaries circulating this week said “better than Blackwell.” The primary data says “cheaper but slower.” Those are opposite conclusions for a latency-sensitive workload.
Before the next accelerator commitment, write down which of those two numbers your workload is actually bought against. Then check that the comparison you are reading reports that one.
How we assure
Introducing the AI Productivity Index for Accounting (Mercor and Ramp)
APEX-Accounting is 160 held-out tasks across ten fictional companies frozen at month-end close. More than 40 accountants wrote them, with a median of 11 years of experience and over half of them from Big Four firms, averaging 13.7 rubric criteria per task. Every model ran every task eight times. Fable 5 leads at 56.4 per cent, then Muse Spark 1.1 at 52.6 and GPT-5.6 Sol at 51.5. That ranking is the least useful thing in the paper. 58 per cent of the tasks were never fully solved by any model on any run. The most consistent model solved just 2.6 per cent of them correctly all eight times. Seven in ten failures were reasoning, not retrieval. Budget sensitivity splits the field. Fable 5 moves from 11.8 per cent at a dollar per task to 55.2 at fifty, while Muse Spark lands within four points of that on about five.
Risk: a single-run score is a capability number, and nobody buys capability. What they buy is a process that gives the same answer on Tuesday as it gave on Monday. Run each task eight times and the headline halves. Every business case that we are shown built on a leaderboard figure is quoting the best of one attempt as though it were an expected value.
Take the process you would most like to automate next. Run five representative cases five times each. The spread, rather than the average, is the number to put in front of your board.
The Fast Gemma Challenge (VIDRAFT, FINAL-Bench)
Entrants had to speed up inference for Gemma-4-E4B on a single A10G with no hardware changes, under a quality gate. The winning verified entry reached 510.58 tokens per second at a perplexity of 2.3930, inside a failure bar of 2.42. A faster run at 535.91 sat at 2.44 and did not verify. Only the results that the organisers re-ran on a private prompt set count as verified. The team published a full manifest alongside it: the vLLM dev wheel, a sliding window of 188, a centroid top-k of 49.
Risk: fastest and fastest-verified are separate claims, twenty-five tokens per second apart, and only one of them survives contact with a private prompt set. Internal optimisation work usually reports the first and calls it the second. A quality gate that the optimiser cannot see is what makes the difference measurable at all.
Ask for the quality gate on your last inference-cost saving, and who ran it. If the team that made the change also owns the measurement, what you have is a speed number rather than a result.