Thirty times the cost for the same task
What an agent costs to run, how well one model marks another's work, and what a vendor has tested. Three numbers people quote as settled.
How we Organise
Input tokens, not output, are where the money went
The same agent task, run twice, can cost thirty times as much. Eight frontier models were measured over a coding benchmark. Which model did the work moved the bill further than which task it was. Asked beforehand what they would spend, the models guessed low every time.
A cost line has to show its parts. Budget from one figure and you budget from the midpoint of a very wide range, on an estimate the model itself put too low.
How we Build
The judges agreed with themselves and not with people
Teams check a model that marks other models on exact-match agreement. That does not correct for the agreement chance produces on its own. Across twenty-one judges and roughly 541,000 judgments, correcting for chance cut the figure by a third to two fifths. Two judges already in production repeat themselves reliably. They still lean on the order they see answers in.
We ask that a model marking a change is never the model that change was made to. Somebody should track how often it and a human reviewer agree. The number teams report is the wrong statistic. Repeatability on its own says nothing about whether the marking is any good. If you have corrected your own judge agreement for chance and found a smaller gap: transform@dromologue.ai.
How we Assure
Four of thirty said what their agent had been tested on
Thirty widely used agent products were indexed against what their makers publish. Most of the safety fields are blank. Four publish an evaluation of the agent itself. Most do not disclose that they are software by default.
A person meeting an agent should be told it is software, should have a route to somebody who can overturn what it did, and should be able to find out what it has been checked against. No firm can meet that last one by writing it into a policy when the vendor has published nothing.