The price of an agent's work is becoming a public measurement
A US national programme is funding a benchmark that prices agentic work per verified outcome, and the field's most-cited evaluator has bound itself to publish what its assurance leaves out.
How we Organise
The benchmark that prices an agent will count only the tasks it finishes
MLCommons will build MLPerf Agents under a 3.5 million dollar, two-year award. The funder is the STRIDE Ventures AI Efficiency Challenge, backed by the US National Science Foundation. The benchmark measures cost, energy, latency and throughput for successfully completed tasks only. Success is verified against hidden validation datasets. Resources are counted across the whole agent stack: harness, model, retrieval, tools, orchestration, failure recovery and validation.
That is the unit we compute for a client from its own records. Our practice states it as a condition:
The cost of one successful outcome is computed from the release’s costs and the team’s eval results rather than assembled by hand.
A public instrument measuring the same unit moves the conversation. The builder is the consortium whose inference benchmarks the industry already accepts. The question stops being whether the unit can be computed at all and becomes what a firm’s own eval record would read under it. This is not the first measurement of the unit. A controlled study this year priced each module of a text-to-SQL pipeline per successful outcome and found no other way to read what earned its cost; a second study’s figures stand open against the same practice, and the two are there for a reader to weigh. The benchmark does not exist yet. What counts as a successful outcome in each workload is still to be written, and that is where the argument over the unit will happen.
How we Assure
The evaluator the field cites most has bound itself to publish what its assurance work leaves out
METR published its standing approach to risk-assessment work with AI companies on 14 October. For each engagement it will describe publicly the scope and duration, the limits on its access and whether they affected its conclusions, and the redactions a company made and whether they mattered. Any other constraint that could have shaped the result is described as well. Conflicts of interest are declared, and METR commits not to be financially dependent on the companies it assesses.
An assurance record that does not say what was left out, with reasons, misleads whoever relies on it. Our practice holds that as a written condition:
The obligations considered and not adopted are recorded with a reason and the person who confirmed it.
The post shows the policy already operating: a redaction summary in its pilot Frontier Risk Report, an in-text disclosure where OpenAI redacted training context, and the access windows published beside its o1-preview results. The terms are self-imposed and nobody checks them. That limits what they prove. What they change is the default: the omitted half of an assessment is now a published obligation at the evaluator whose reports the rest of the field reads, rather than something a client must ask for case by case.