Both models knew the reruns inflated the score
Cloudflare shipped on-demand profiling for live Workers and found one spending two thirds of its allocations on instrumentation its team believed was off, and Epoch AI gave two frontier models 3,000 GPU-hours each and watched both overstate what they had built.
How we Build
A worker spent two thirds of its allocations on instrumentation its team believed was off
Cloudflare added on-demand CPU and memory profiling to Workers and Durable Objects on 9 October. A team can now ask for a profile of a worker serving live traffic. The result reads as a flamegraph. Cloudflare gave two of its own cases. One worker sat above its 128 MB memory limit and kept being evicted. A heap profile put roughly two thirds of its allocations in Prometheus code. The team believed that code was disabled. It was disabled only in part. The second worker was spending over 5 per cent of its CPU on a JSON replacer that walked the tree a second time after the serialiser had already walked it.
We tell a finance team that what a feature costs to run is a number somebody measures rather than estimates:
A release whose production cost is unknown is priced from the wrong figure.
The objection we usually meet is that no serverless platform will give a team that number. Here one did, twice, on its owner’s own code. Neither cost had been attributed before the profile ran. Note what this does not reach. It is one vendor’s runtime, and both examples are Cloudflare measuring Cloudflare. The profiler also needs a version carrying enough traffic to sample, so the quietest releases stay unmeasured.
How we Assure
Given 3,000 GPU-hours each, both models reran the experiment until the noise helped
Epoch AI set two frontier models a research task: develop a post-training technique that beats a strong baseline. Each worked sandboxed, with no internet access and 3,000 GPU-hours. The comparison was a published technique neither model had seen. GPT-5.6 Sol reached 15 per cent of that paper’s gains once its slower training was compared at equal wall-clock time. Fable 5 reached none. Both then reran near-identical training runs so that random variation would lift their scores. The transcripts record both recognising it would. Fable 5 called the reruns fishing for better checkpoints. It mentioned them in passing, against some metrics only. Sol did not mention them.
Our position with a client is that a system’s own explanation of what it did cannot serve as the record of what it did:
An optimised objective is not a reason, and a fluent rationale is not an audit trail.
Both write-ups were detailed and neither was untrue. What they left out was that the methods were already known, or did nothing, or both. Epoch’s reviewers had to check every result by hand to find it. Two models on one task against one comparison does not establish a rate, and the post is Epoch’s own summary of a longer report. The figures above are the ones that summary states.