Anthropic redacted most of the evidence behind an independent assessment
Three organisations put a number on their own work this week, and in each case the party with an interest in the number also decided how much of it anyone else could see.
How we Organise
By June 2025 Docusign had 49 per cent of its engineers using AI coding tools and an 18 per cent productivity gain that had stopped moving. It added a six-person central team to write guardrails and run workshops. Then it put around 75 AI Multipliers inside individual delivery teams. Each was responsible for deploying the central team’s decisions locally and for reporting back what was missing. Adoption reached 65 per cent by October 2025 and now covers all of its more than 1,100 engineers. Lines of code updated per developer are up about 52 per cent.
We tell a client that whoever ships a capability owes a lesson to people outside the team that shipped it. Our practice says it plainly:
Shipping a capability creates an obligation to teach somebody outside the team.
Docusign’s chief technology officer makes that case about his own firm. He says many AI programmes start well and then fade, and that this one did not, because the Multipliers kept feedback moving in both directions. The code-volume figure is the one to treat carefully. The article says that measure does not establish faster delivery or better software, and nothing in it separates what the Multipliers did from what a year of familiarity did.
How we Build
A mobile test suite went from 50 per cent reliable to 98 by reading the screen instead of the code
Shopify’s mobile end-to-end tests had run since 2023 on Appium, finding elements by the identifiers developers had put in the code. Tests tapped before screens finished rendering. Engineers patched round that with fixed pauses that passed on a laptop and failed in continuous integration, and the suite ended up removed from pull-request checks altogether. The rebuild finds elements the way a merchant does, reading screen text with PaddleOCR and matching screenshots against the firm’s own design-system graphics, behind an interface where every step must declare what it expects to see.
This is about what a boundary depends on. We write it as a condition a client can check:
A context owns the value it produces, not the vocabulary it uses.
Shopify’s own account says why. A passing test proved a node existed in the component tree, and proved nothing about whether a merchant could see it or use it. The limits here are Shopify’s too. It measured its own suite and nobody read the figure independently, no run cost is published for matching screenshots against a direct lookup, and the old identifiers survive as an opt-in fallback on screens where generated content defeats the visual match.
How we Assure
An independent assessment of one lab’s acceleration was published without most of its evidence
METR spent 12 August to 29 October 2026 inside Anthropic, with access to internal metrics on pretraining, reinforcement learning, training compute and benchmark scores for unreleased models. It concluded that Anthropic is now improving model capabilities on well-specified tasks at around twice its March 2024 to October 2025 rate, with Anthropic’s own capabilities index growing at an estimated 1.9 times the earlier pace. Anthropic held the right to redact anything non-public and used it. METR writes that Anthropic removed a majority of the evidence METR found most informative, and says the report does not achieve transparency about evidence.
We ask a client to name two different people. One can alter a record after the fact. The other controls how much of that record an outsider doing the checking ever gets to see. The statement we hold is this:
Who may change the stored logs and who may change what a reviewer is shown of them are stated separately, and the agent whose actions they record can change neither.
The condition that decides it reads each agent’s own record, and the only party it can rule out is the agent itself. So a firm can hold a complete and current entry for every agent it runs and still decide what an assessor may publish about it. This is not the first source to bear on that practice. GitHub’s telemetry change raised it on 24 September, and METR’s own note on agents concealing their misbehaviour added to it on 7 October. METR calls its risk conclusions subjective, its reading of the bottlenecks moved as more material arrived, and redacting commercially sensitive material is lawful and ordinary. We would rather hear the argument than settle it here, so write to transform@dromologue.ai.