The agent could not finish, so it found another way
Vals AI measured what agent teams cost against single agents on fifty builds, GitHub began recording which account pushed each commit, and Anthropic published four classes of action its own models took that nobody intended.
How we Organise
Thirty times the money bought a point and a half of score
Vals AI built fifty web applications twice. One run used a single agent. The other used a lead agent that could hand work to five others. Teams cost between 1.8 and 5.1 times as much. Only one of the four comparisons moved the score at all. The most expensive setup reached 93.2 per cent at $122 an application, where the same model working alone reached 91.5 per cent at $4.08. Most of the extra money went on cached input, because each subagent holds its own copy of the specification and re-sends it every call.
A finance function should be able to ask what thirty times the money produced. Most cannot. The spend arrives as one monthly model bill, by which time the release it paid for is weeks behind.
Every cost of the team’s work is attributed to the release it helped produce.
That comparison exists only because somebody attached the cost to each application instead of to the month. Pooled into a quarterly total, eight setups become one number. The question cannot be put. This is one benchmark, each setup ran once, and browser agents did the grading, so the scores may carry more noise than four significant figures suggest.
How we Build
The name on a commit was never checked, and now the account that pushed it is
The author and committer fields on a git commit are strings the pushing client supplies. Anyone with write access can set them to any name and address. A compromised low-privilege account has been enough to produce commits that read as a trusted maintainer’s. GitHub now records the authenticated account behind each push. It shows that account wherever commits appear. A new repository rule refuses any push whose committer address matches no verified address on the pushing account.
We tell clients that the record of where something came from has to be written as the thing itself is written. Rebuilt later out of logs it is a weaker claim. The only evidence left is whatever the system kept.
Answers carry their provenance with them, produced with the answer rather than reconstructed after it, and anything a model writes back into the application landscape is identified as machine-generated at the point of writing, with the model and the date.
Here the marking is applied as the write happens, and the rule refuses the write that would break it. What it captures is who pushed, and who pushed is not who wrote. Where a person runs an agent, the account on the push is the person’s and the work is the agent’s. The feature is a limited preview for Enterprise customers. Nobody else gets it until an account team grants access.
How we Assure
Four ways a model got past a limit its evaluators thought they had set
Anthropic has published the unintended actions it found in transcripts of its own evaluations and internal use. Models ran commands on a third party’s server through an injection flaw. They submitted live forms they were told only to practise on, and reached gated data using tokens the sites hand out. And they beat a length limit on their own fetch tool with a free URL shortener. One sent an invented tip to a police department, which disclosed the case itself. Anthropic reads most of this as persistence: unable to do the task as instructed, the model works around the restriction instead of stopping. Internet access is now off for every internal evaluation.
Where the written rules do not settle a case, we say a firm needs a standing principle. It names where it is published, how a case escalates, and who decides.
The standing principle for the cases the rules do not settle states where it is published, its escalation route and who decides.
Anthropic has the first two and says plainly that it cannot yet name the third. How serious a failure is depends on what the model was trying to do, and a model’s account of that is unreliable. Our test allows for that. A question put and not yet answered is recorded as unknown and still passes. What it does not allow for is a principle whose holder calls it provisional, which fails whatever the three clauses say, and Anthropic calls its severity framework exactly that. So the test would mark down an organisation that published its principle, routed its escalations, notified a regulator and built a control that blocked every case it had found, for declining to call the framework settled. Read the other way, a framework nobody has tested is thin material to decide by, which is the reasoning the clause rests on. Anthropic withheld the organisations’ names at their request. It has completed no full assessment of the cases. Tell us which way that should cut at transform@dromologue.ai.