It came back in the right format
How we organise
The Fabricated Front: Generative AI and the Opacity of Workplace Performance (van Nuenen, Sachdeva and Chopra)
Tom van Nuenen, Pratik Sachdeva and Sahiba Chopra asked a narrower question than most. They were not asking what generative AI does to jobs, or what it does to output, but what it does to the signals colleagues read off one another when they decide who is good. They read 1,250 interview transcripts from Anthropic’s AI Interviewer dataset, and they name five such signals. Two of them matter here. Provenance is who can stand behind a piece of work. Investment is how much effort the work reflects. What the transcripts show is an asymmetry rather than a collapse. Professionals defend provenance, because their name is on it. Around investment they are candid to the point of comfort, describing the work they now hand to a model while the delivered thing looks as it always did. The authors do not treat this as a new dishonesty, but as the conclusion of workplaces already organised around output, where what is delivered has stood in for the effort behind it for years. Separating the two now takes almost no work.
We have argued that a person’s level should rest on work they have shipped, because delivered work is something another person can go and check. This is a reason to doubt that. If what is delivered no longer reflects the effort behind it, the test still passes, on someone whose judgement nobody watched. The view holds only where what is watched is the judgement rather than the artefact, and that is not what we have said. There is a limit to how far the finding travels. It is interview data, and it is people describing their own conduct, so it measures nobody actually being deceived. A reader could hold that good managers discount delivered work already and read the questions a person asks instead, in which case nothing has changed. If the finding is right, we stop telling a client that shipped work evidences capability, and say instead that it evidences capability where somebody watched the judgement. A manager would then have to watch the work being done rather than read a record of what arrived.
How we build
A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations (Mahmud and others)
Hasan Najib Mahmud and colleagues took working codebases and rewrote them into a form that does the same thing. They reshaped control flow, added dead code and renamed identifiers. Nothing the compiler would notice. Then they set the agents to work again. They ran two scaffolds, mini-SWE agent and OpenCode, each on one of four frontier models, over instances drawn from SWE-bench Verified and SWE-bench Pro. Every instance ran several times on the untouched code and on the rewritten code. That is what separates the effect of the rewrite from the agents’ own variance. Most configurations degraded a little. The worst lost 6.7 percentage points of mean resolve rate, and six of sixteen moved significantly. The finding with the longer reach is the one about ordering. No ranking of models by robustness held across both scaffolds. Qwen was among the most robust under mini-SWE agent on SWE-bench Verified, and among the least robust under OpenCode. Part of what these numbers measure is the scaffold rather than the model.
The tests we ask of a skill are written before the skill is built, and some of them are properties that must hold whatever the input. Semantic equivalence is one of those properties. The paper does not argue with that. What it shows is how little a passing test proves on its own, because the same test reverses under a different scaffold. What we have left out is that a result should record the scaffold it was produced under, and should not be read past it. A reader could take the results the other way. The drops are single-digit, most configurations were not significant, and someone could reasonably conclude the effect is real and small. If it is larger than that, the claim that expires is ours, that a green suite evidences a robust skill. Every client who has changed scaffold since their tests were written holds a result that stopped being true that day, and a board that still shows green.
How we assure
Outcome Monitors: Recovery Affordances for Silent Tool Failures (Panthi and Abdelfattah)
Sugam Panthi and Rabab Abdelfattah start from a distinction that is not available to most agents. A tool call that times out is a visible failure, and the agent routes around it; a cached error page arrives in the expected format and is read as fact. So is a negative price. The failure is silent because the format is right, and the format is most of what a model has to go on. Their monitors sit on the return path. They check each return against outcome contracts, some mined from earlier traces, some taken from public schemas. Where a return breaks its contract the monitor preserves the result and issues a receipt naming the property that broke and the recovery tools now available. In frozen, prespecified evaluations with injected failures, ToolMaze completion rose from 10.9 per cent to 28.1 per cent. That held across four models in two provider families, replicated in a third, and on tau-bench retail two tiers improved by 14.0 and 12.0 points. The authors then ran the ablation that decides what is doing the work. Stripping the recovery tools out of the receipt removed the gain. Putting them back restored it. Varying the receipt’s detail and its timing changed nothing measurable.
Checking what a tool returns, before it reaches the model or anything downstream, is something we already argue for, and the paper bears it out. The ablation then goes further than we do. Checking the return is not the part that helps. Naming what the agent may do next is the part that helps, and that is a larger claim than ours, and a more practical one to act on, since a receipt that names no recovery tools changes nothing about what the agent does next.
Their own limit belongs beside the headline figure. On a suite transcribed from a published incident taxonomy, detection outside the mined vocabulary fell to 46 per cent. A monitor built from your own traces detects the failures already in those traces, and little beyond them.