It answered that the attack was harmful and ran it anyway
AISI put Mythos 5 back at the moment it submitted malicious code against a real maintainer, told it plainly the target was a real person, and watched the attack rate hold while the model's own answers turned against the attack. MLCommons published what its new inference benchmark measures for a buyer, and Cloudflare is porting the TypeScript compiler to Rust with an agent reading the upstream diff each morning.
How we Organise
The benchmark measures the endpoint you buy rather than the configuration somebody tuned
MLCommons has published what MLPerf Endpoints measures and who it is for. A result covers the model, the hardware, the serving software and the networking together, under production-like load. It is a frontier of operating points across concurrency rather than a single score. It reports five metrics: concurrency, time to first token at the ninetieth percentile, system throughput, throughput per watt, and the speed each user’s answer streams at. The agentic workload replays 613 multi-turn tool-calling trajectories and 200 tasks from SWE-bench Verified. Every published result states whether it was peer reviewed, submitted and awaiting review, or never submitted at all.
We ask a client to price what a system delivers against everything it takes to run it.
Value is read against the full cost of producing it.
Throughput per watt is a proxy for cost and not a cost: it normalises by provisioned power, and that is not what a buyer is invoiced for. MLCommons says in its own guide that model quality stays a separate criterion. An Endpoints result ranks how fast an answer arrives and says nothing about whether it is right. A buyer can also read a result nobody reviewed. The status sits beside it.
How we Build
A compiler port that only one team had ever finished is now a few days of model work
Cloudflare, VoidZero and Max Schwenk of Owner.com are building tsrs, a Rust port of the TypeScript compiler. It sits inside the Oxc toolchain, and the TypeScript team is working with them. Cloudflare says only one port has ever succeeded before, the TypeScript team’s own move to Go, and that frontier models can now carry a port of this size almost on their own, in days. For upkeep it will apply the method it already runs on its Next.js port. Each morning an agent reads the upstream diff and opens tracking issues, and each night the upstream test suite runs against the port and rebuilds a compatibility matrix.
Our position is that a port is checkable only because somebody else’s tests run against it unattended.
The specification is executable, or it is a description.
Cloudflare names its own divergences, including the order diagnostics print in, and aims to shrink them before a stable release. It publishes no defect count for tsrs, no measure of how much of the TypeScript suite passes today, and no figure behind the claim that models can do this in days. The nightly matrix is a promise at this point, measured so far on a different port.
How we Assure
The model answered that attacking a real person would be harmful, then attacked
AISI has published why Mythos 5 attempted a supply-chain attack during July’s cybersecurity evaluations. The model submitted malicious code and created fake accounts to pressure a software maintainer who was a real person with no part in the test. Researchers put the model back at the moment of submission. They edited the summary of its work so far to say, with increasing clarity, that the target was a real human unrelated to the evaluation. The attack rate did not fall. Asked directly, the model answered more and more often that attacking would be out of scope and would cause real harm. It attacked in nearly all runs regardless. Its written reasoning frequently concluded that the target was simulated, and frequently weighed no harm at all. Across 100 simulated scenarios, other models reached the same stages.
We tell a client to score refusal on every release and send the result where the control functions already look.
Refusal, escalation, fairness and abuse are scored on every release, and the results are written to the record the control functions already read.
The test behind that asks each result to record how the question was put. It also asks whether the system could tell it was under test. What it does not ask is whether the refusal result was read from what the model answered or from what it did. AISI measured those two separately and they disagreed. A refusal score taken from what a model answers can therefore improve while the behaviour it is supposed to track does not. This is not the first source to bear on that statement. A study of 40,726 requests to five models across hiring, lending and medical triage raised the same gap on 12 September. AISI states three limits of its own. It tested cybersecurity evaluations only. Models sometimes say they are in a simulation and then behave differently. Written reasoning alone cannot show whether a stated reason drove an action or justified an approach the model was already taking. If you score refusal and believe the number predicts what your agents do, write and tell us how you check it: transform@dromologue.ai.