Roblox fed its own review history back as rules
Three firms published what they changed and what followed. One moved human judgement from individual changes to the rules a machine reviews against, one spends its tokens before anything ships, and one bought a dependency control to stop having the argument.
How we Build
Comments a machine wrote are taken more often than comments the engineers wrote
Andrew Swerdlow, vice-president of engineering at Roblox, says the firm mined years of its own code review comments into rules it calls exemplars. Those rules now guide both generated code and machine review. Acceptance of machine-written review comments runs at nearly 70 per cent. Comments written by people run at 50 to 55. Human judgement moved off individual changes and onto the rules themselves. About 5 per cent of pull requests now merge with nobody reading them, after machine review, programmatic safety checks and a risk assessment. Swerdlow credits strong testing, production canaries and automated rollback. Caching and shared context cut the cost of evaluating one rule by 90 per cent.
We ask a client to count how many of the machine’s suggestions its engineers actually take. The checks whose suggestions keep getting refused are then held back.
Review is sized and routed so that a machine checks what it can before a person sees anything, including against the design principles the build must honour; findings reach a person only where the author disputes them, and the share of the machine’s findings the author accepts is measured, with findings from a check below the team’s threshold held back.
Roblox is the first named firm we have seen keep that count and route on it. It is not the first source on the question. In September a study of 31,073 review-and-feedback pairs across 239 repositories, using CodeRabbit, put acceptance at 36.4 per cent and rejection at 56.3. It argued that a team can route review correctly while its authors absorb a queue that is mostly invalid. Roblox reports nearly double the acceptance. What it credits is the review history, not the model. These are figures an executive gave in an interview, with no published method and no denominator for the 5 per cent. A comment an author takes may still be wrong. Acceptance is not correctness.
How we Organise
The tokens were spent before anything shipped
Ryan King, chief information officer of the hydraulic parts maker Brennan Industries, builds ERP applications with Claude Code and OpenAI Codex. He takes requirements from a colleague in person and has a mockup back 45 minutes later. The agent then writes SuiteScript, which is deployed onto NetSuite and runs there. Nothing is left running on the AI platform. No token is spent when a transaction is served. He puts the rate at up to five projects a day where each previously took a week or more. He has an answer to the complaint that agents are expensive. The complaint is about leaving the work on the platform that built it.
We tell a finance team that a release’s cost has to break into parts, and we name three of them.
A release’s cost is stated in its identified parts rather than as a single figure.
King’s arrangement satisfies that. It also shows what the three parts cannot say. A model or token row reads the same whether the spend happened once or recurs every time the thing is used. At a hundred times the volume the two cost a firm very different amounts. The account has limits of its own. It rests on one firm, with no token figures either way, from an interview at a vendor conference whose host paid the journalist’s travel. King’s caveat is that somebody has to know the ERP well enough to instruct the agent. If you price agent work for a living and read this differently, write to transform@dromologue.ai.
How we Assure
A dependency control was bought to stop the team having the argument
Dan Conn, staff platform security engineer at Trustpilot, calls agents the ultimate insider threat and asks that the word think be dropped from the description. An agent carries no malice and will not stop, because reaching its goal is all of its behaviour. On dependencies he refuses always taking the newest version. Doing so would have left the firm open to about twelve zero-day vulnerabilities this year, he estimates. He settles on a version before the latest but not far before. He gives his reason for buying a supply-chain control plainly. It takes the problem out of his team’s hands so the discussion need not happen. He sets severity against his own firm’s data rather than a general standard.
Our risk principle holds that a control heavier than the risk it reduces drives work out of sight.
Where a control costs more than the risk it reduces, people use systems nobody can see.
A security engineer buying a control to remove an argument is that principle reached from the other end, before anybody had worked around anything. There are no figures for Trustpilot itself. The twelve zero-days is an estimate of exposure avoided rather than a count of anything that happened. Nothing measures whether the removed friction changed what engineers did. The quantities given at the event were the vendor’s own, at its own conference.