← AI Feed
AI Feed

The cheap model with a harness came out cheaper

A cost-optimised coding model went from three accepted runs in ten to ten out of ten once it was given contracts and executable checks, and the harnessed cheap model cost less per run than the frontier model working unaided. Anthropic replaced git outright in five months with seven people, and a trailing dot in a filename let an assistant rewrite the file listing its own tools.

How we Organise

The cheaper model turned out to be the dearer one per accepted application

Naresh Jain built the same product-discovery service ten times over with each of two models. He scored every finished application against 178 acceptance tests it had not seen. GPT-6 Luna is the cost-optimised model. On its own it was accepted in three runs of ten. Given machine-readable contracts and executable checks to work against, it was accepted in ten of ten. That second configuration cost about 0.273 dollars a run. The frontier model working unaided cost 0.406 for the same ten-of-ten result.

We tell a finance team to price a model on the work it gets accepted rather than on the attempts it makes.

A cheaper answer that is wrong more often costs more.

Divide by accepted runs. The unaided cheap model lands at roughly 0.287 dollars an accepted application, against 0.273 harnessed. The measurement supports that by about five per cent, rather than by the third the headline offers. It covers one task and ten runs a cell, and the costs are estimated from token use rather than taken off a bill. The reverse case is sharper. The same harness on the frontier model cost seven times more and changed nothing.

How we Build

Seven people replaced git and finished the migration in five months

Anthropic’s source-control team has published how it replaced git with a system of its own. Git assumes a client working offline, holding a complete copy. Nearly all of Anthropic’s clients are agents and jobs that are online by definition. Seven people built the replacement and a code-review tool beside it, and finished the migration in five months while daily merge volume rose tenfold. They report a running footprint 200 times smaller, and checkouts falling from minutes to seconds.

Our position is that a core system is replaced a slice at a time.

The core is replaced in slices, never rewritten wholesale.

Anthropic says the years of iterative migration this would once have taken are what agentic development let it skip. It is one firm and one system, reported by the people who built it, with no counterfactual attempted. They name two limits themselves. The design suits only enormous monorepos, and their internal token spend exceeds what another business would want to spend. Source control is also developer tooling rather than a core carrying live business transactions. If you have replaced a core either way and have the dates, we would like to hear it, at transform@dromologue.ai.

How we Assure

A trailing dot let the assistant rewrite the file listing its own tools

Microsoft patched a flaw on 13 October that Johann Rehberger had reported in August. The rating is important, at a severity of 8.8. The check stopping GitHub Copilot from editing its own settings compared filenames as text. Windows drops a trailing dot when it writes a file. So a name ending in one passed the check, while the write landed on the protected file. From there the assistant could add a command to its own list of tool servers and have Visual Studio launch it. Rehberger ran the same sequence from outside, using content the assistant had only been asked to explain.

We ask a client to assume an agent can be turned by the material it reads.

An agent can be compromised by what it reads.

The hijack arrived as file content. The control that failed was written specifically to hold the agent. One researcher found it, in one product family. The defect sits in a vendor’s compiled library rather than in anything a client set up. The affected code is shared with SQL Server Management Studio.