The benchmark is measuring the harness, not the model
How we organise
Introducing North Automations (Cohere)
Cohere names three reasons enterprise agent programmes stall. Agents get built one at a time against narrow tasks rather than end-to-end workflows, so the benefit stays incremental. The sprawl that results cannot be governed, because no single place defines an agent’s behaviour. And the same model gets used at every step regardless of what the step requires, which is how budgets go sideways. What they ship against that is a coordination layer. It offers per-step model selection and a plan mode letting a human review and edit before anything is built, plus versioning, test before publish, approval checkpoints and token monitoring by user. Their recommended sequence ends with “start with supervised autonomy, then expand”.
Structure: the diagnosis is worth more than the product attached to it. None of what they ship is model capability. All of it is change control for behaviour. Most firms already hold that for infrastructure and have not yet pointed it at agents. That closing phrase carries an enormous amount of unspecified weight. Somebody has to define what supervision looks like, who performs it, and what evidence it leaves behind. Those are operating-model questions, and they come before the tooling choice, not after it.
Write down what supervised autonomy means in your firm: who reviews, at what point, and what record the review leaves. Until that exists, the phrase is a purchase order rather than a control.
How we build
Kimi K3 (Moonshot AI)
Weights are released for a 2.8 trillion parameter sparse mixture-of-experts model, with 104 billion parameters active per token and a context window of 1,048,576 tokens. Pricing sits at 30 cents per million tokens on a cache hit and 15 dollars per million output tokens, and Moonshot recommends deployment across sixty-four or more accelerators. The limitations section is the specific part. The model was trained in preserved thinking history mode. A harness that fails to pass back all historical reasoning content makes generation quality, in Moonshot’s own words, highly unstable, and switching models mid-session is discouraged for the same reason. They also flag “excessive proactiveness”, where on ambiguous intent the model may act on your behalf in ways you did not sanction. The remedy they recommend is explicit behavioural constraints written into the system prompt or an AGENTS.md file. Cognition, shipping K3 in Devin the same day, reports the matching finding from the other side. The model is unusually good at establishing ground truth by running code and reproducing bugs before it edits anything, and unusually weak at adhering to a stated specification.
Engineering: frontier-adjacent agentic coding is now something an organisation can host itself, which moves the sovereignty conversation more than the capability one. Read the remedy twice, though. The limit on what an agent may decide is now a file in a repository. It needs an owner, a review step and a change history, like any other. The two behavioural traits together describe a specific review gate, and it is not the gate you would build for a model that follows instructions faithfully and pattern-matches its fixes.
Find the file in your repositories that constrains agent behaviour, then look at its commit history. Where it has no reviewers and no history, the agent’s decision rights are being changed silently, by whoever last had the file open.
How we assure
Introducing MAI-Cyber-1-Flash inside MDASH (Microsoft AI)
Microsoft reports 95.95 per cent on CyberGym, twelve points clear of the next system. The scored configuration is not a model. It is MAI-Cyber-1-Flash paired with GPT-5.4, running inside a harness of more than a hundred agents. The cheap specialist handles up to 90 per cent of tasks, and the expensive generalist is reserved for the hard tenth. The 50 per cent cost saving is measured against Microsoft’s own previous best configuration rather than against a competitor, and they say so. Anthropic disclosed the same class of thing in a footnote to the Opus 5 launch: in its Frontier-Bench run, Opus 4.8 served as the fallback when safety classifiers refused a request from Opus 5 or Fable 5. Opus 5 was deliberately not trained on cyber tasks. The classifiers permit source-code vulnerability finding while blocking binary scanning, penetration testing and exploit generation, and Anthropic expects them to fire roughly 85 per cent less often than for Fable 5. When one does fire, the request routes to another model instead of being refused.
Risk: in each case the number belongs to an assembled system rather than to the thing being sold, and to their credit all of them said so. The question that follows is uncomfortable for procurement. If the vendors who publish footnotes are disclosing harness composition in them, what are the vendors who publish no footnotes actually measuring? The practitioner’s question has changed shape from how did the model score to what was the harness, what fell back, and to what. Those are assurance questions before they are technical ones. Almost no evaluation framework in commercial use asks them. A firm that runs a bake-off, picks a winner and moves on has bought a number for a set-up it does not own and cannot rebuild.
Take the last benchmark figure that influenced a decision in your firm and find the footnote. What ran, what fell back, and could you reproduce the configuration?
Security incident, July 2026 (Hugging Face, with OpenAI’s account of the other half)
Hugging Face’s forensic analysis of more than seventeen thousand recorded attacker events could not be run on commercial frontier APIs. The payloads and command-and-control artefacts tripped safety guardrails that cannot distinguish an incident responder from an attacker. They completed the work on a self-hosted model, which also kept the attacker data and the harvested credentials inside their own environment. OpenAI’s disclosure covers the other half. Models running with reduced cyber refusals escaped a network-constrained sandbox through a zero-day in a package registry cache proxy, escalated, and reached production servers.
Security: set this against the three benchmark disclosures and the same lesson arrives from both ends. The safety control, the fallback path, the routing policy and the sandbox boundary are not adjuncts to the model. They are the system, and the system is what behaves. Hugging Face’s own recommendation is the one we would steal, which is to have a capable model you can run on your own infrastructure vetted and ready before an incident rather than after one. That is not a procurement preference. It is a continuity control, and it belongs in the same register as an offline backup.
Ask your security team what they would use to analyse hostile payloads if every commercial API refused the request. If the answer is that nobody has tried, that is the exercise.