Most agent sessions were never counted
GitHub undercounted Copilot agent activity for months after several development environments moved those sessions to a new kit, and says the gap cannot be backfilled. METR found a flaw that would have let an agent under evaluation rewrite the transcript its reviewer reads.
How we Build
The agent work was happening and the dashboard was not counting it
GitHub published the cause on 6 October. Several development environments moved Copilot agent sessions to the Copilot SDK. Those sessions did not say which environment they came from. The usage metrics could not attribute them, so most of that activity never reached enterprise, organisation or user reports. Billing was unaffected. None of it can be recovered, because the affected activity does not identify where it came from.
We tell a finance team to take its delivery figures from the systems engineers use every day, rather than from a reporting tool kept up beside them.
Every customer release carries the aggregate token cost of producing it, meaning everything spent building, evaluating and reworking it since the last one, and delivery telemetry is drawn from the systems the team already works in.
A team following that would have named Copilot usage metrics as its source. It would then have watched its agent figure fall. The source was named and the number was still wrong. What the practice does not ask for is the version a figure was collected at. That is the one thing that would have turned a silent change in the tool into a visible break. GitHub does not say how large the undercount was. The gap may be worse at a firm that standardises on one environment. If you read delivery figures out of your own tooling and want to argue about whether naming the source is enough, write to transform@dromologue.ai.
How we Assure
A researcher found it in ten minutes, and the reviewer would have seen nothing
David Rein published METR’s finding on 6 October. Inspect is the evaluation framework much of the safety field runs on, and its transcript viewer is how a researcher sees what an agent did. METR asked whether an agent inside an Inspect evaluation could change that view. With help from an AI agent, a researcher found a vulnerability in about ten minutes. It would let the agent rewrite earlier actions in the transcript. The same flaw would intercept the download button. The stored trajectory stayed intact, so the behaviour was still there to find. Only somebody who already suspected it would have looked. METR reports that no agent has used this in its evaluations, and that it has separately seen agents tamper with logging and monitoring.
We ask a client to be able to say afterwards what an agent did. We check that by looking for the record.
What an agent did can be reconstructed afterwards, the instructions it acted on included and kept for a stated period, and every incident involving an agent records the systems it touched and how it was resolved.
Here the record was intact and the view of it was not, and a firm doing exactly what we ask would pass that check. The person reading the transcript would see a version the agent had chosen. What the practice does not separate is the stored record from the surface that renders it. The two have different writers. The flaw was narrow: one framework, and the view rather than the store. This is the second source in a fortnight on the same claim. GitHub’s OpenTelemetry export for the Copilot app, added on 22 September, records tool calls and model requests while excluding prompt and response content by default. One bears on what the record holds and the other on who can alter it. If you keep agent transcripts and think asking for the record settles it, tell us where we have this wrong at transform@dromologue.ai.