The control we named is not the one that held
How we Organise
Two systems a scorecard called identical, and a third more review between them
Veronica Chatrath and seventeen colleagues built a qualification procedure and ran sixteen agent systems through it on 750 held-out cases of a clinical audit. The procedure does not ask how well an agent works alone. It fixes a reliability target and searches the review policies available. It returns the cheapest one that reaches the target, then tests it on cases it has not seen. Working unaided, GPT-5.4 answered 72.8 per cent of cases correctly and Claude Sonnet 5 answered 72.5. To reach a reliability of 76 per cent, GPT-5.4 had to send 39.2 per cent of its cases to a person and Sonnet 5 had to send 29.6. The authors put the difference down to confidence rather than accuracy, meaning how sharply each system sorts its right answers from its wrong ones. They also assume a person is correct 90 per cent of the time. On that assumption no policy took the combined system above 90 per cent reliability, at any level of review.
We tell clients that the cost of an agent is the whole cost of a good outcome, and that a partial cost will mislead whoever reads it. Most of the time we are arguing about tokens. This is the same argument reached from the other end, and it is the sharper one. A buyer given the first pair of numbers sees two systems within a rounding error of each other. A buyer given the second pair sees a third more review on one of them, which is a person’s week, and it is the number a budget has to carry.
One clinical audit, one team’s implementation, one family of review policies. The percentages belong to that workflow and are not a rate anyone can quote. Somebody could read the whole result as a point about how well a model knows when it is wrong. That is true, and narrower than what we take from it.
How we Build
The part of a tool interface everybody writes down is the part that broke least
YiShan Zheng, Yuan Wu and Yi Chang wanted to know where tool calling breaks rather than how often, so they built a benchmark that damages one stage at a time. Four families of damage follow the pipeline: the tool interface itself, the user’s intent, what the tool returns, and the runtime around it. Each failure is attributed to a stage. A wrong answer separates into choosing the wrong tool, misreading the schema, binding the wrong argument, or mishandling what came back. The benchmark runs to 15,456 single-family cases across seven models and fourteen kinds of damage, using sixteen tools drawn from a local set of forty so that nothing moves underneath the measurement. Averaged over the models, damaging the tool interface was the mildest of the four families, leaving 0.918 of clean performance. Damaging what the tool returns was the worst family on every model tested. Combining two families produced failures that neither family predicts alone.
Our position is that a team should keep a register of every interface a machine consumes. It records the protocol, the written description a model reads, and how many tools a caller is choosing between. The register is a good idea and we are not withdrawing it. What we have also said, and should not have, is that a team keeping it can see the state of its machine-facing surface. Every field in that register describes the interface, and the interface is the family this measurement found matters least. What the tool sends back has no field at all.
So the register is not wrong. It is looking at the wrong end. A team can hold a current entry for every tool it publishes and pass our test in full. Its largest failure mode sits somewhere the register does not reach.
There is room to disagree. Sixteen tools in a fixed local set is a controlled setting rather than a production one. It was chosen so that live services could not move the results, and an estate with hundreds of tools might well find selection dominating again. The four families are deliberately not ranked finely against each other. The reading above needs only the gap between the mildest and the worst, which they do claim. Teams whose tools return well-typed values have an objection. To them this is a problem about other people’s tools.
If it stands, we stop telling a client that a complete register evidences a sound machine-facing surface. The register grows a column for what a tool may return, and a second for what the caller does when the shape is wrong. That matters most to whoever signed off the interface work as done.
How we Assure
Ten ways of asking for the secret were refused, and the eleventh wording got it
Md Habibur Rahman and Jaeho Kim gave an agent a secret, a set of mock tools and a web page written by an attacker. Then they measured how often the secret left, against a matched run where the page was clean. Across six models, ten overt classes of instruction to leak it were refused outright. Rewording the same request as a required integrity signature, a configuration field, or an address that looks like a trusted one took one model from never leaking to leaking every time. An ablation locates the mechanism. Removing the confidentiality rule altogether left the overt attempts at nothing, and moved the reworded ones only from 31.9 to 38.1 per cent. The model is not being talked out of a rule; it cannot tell an instruction from the page it is reading.
Then the defences, which is where this stops being a paper about attacks. Keeping the page in a separate input channel from the model’s own instructions left 38.8 per cent of reworded attempts leaking, against 21.2 with no separation at all. A published fine-tuning defence left 32.5 per cent. It was checked against the attack it was built for and stops that one, so this is a failure to generalise rather than a broken setup. A guard that reads outbound text and normalises it lost to an encoding nobody had written it for. Two checks that never look at the payload closed the leak entirely. One refuses any destination not on a list. The other splits the agent, so the part reading the hostile page never holds the secret.
We say that a team keeps the model’s instructions apart from the content it retrieves, and that a suite covering the direct, indirect and delayed cases proves it. We may be wrong, and this is the first measurement we have seen that puts a number on it: under it, the separation sits in the same twenty to forty per cent band as doing nothing. Nor are the three weaker defences ranked against each other, and we should not rank them either. The finding is the gap between all three and the two that worked.
The suite half fails for a second reason, and a client would feel this one sooner: the overt cases a standard suite carries are exactly the ones every model refused. Run that suite on every change and the result comes back clean while a rewording walks through, and that rewording costs the attacker one paraphrase. Paraphrasing a mechanism that already worked succeeded 96 per cent of the time.
Room to disagree, and a lot of it. This is one synthetic lab with an invented secret, a single step and mock tools. The learned defences all ran on one small open model under one harness. Where legitimate destinations are open rather than closed, the allow-list result does not transfer at all, which the authors say plainly. Somebody could hold that a production harness closes the gap, with real tools and several steps. Nothing here rules that out.
What we would have to stop saying is short. Separation is not the control, and a passing suite is not evidence that it holds. Every client we have put on that pairing has a clean board and a path out of the building.