Four million entries, thirty thousand read
Two engineering accounts where the number a system reported was accurate about the wrong thing: a cache sized from its repository rather than its traffic, and a sync that healed drift and called it a release.
How we Build
Thirty thousand of four million entries were ever held in memory
Uber profiled its Go services. Garbage collection was taking more than a fifth of the processor time in some of them, and translations made up about 70 per cent of the objects being allocated. It rebuilt the translation cache to keep every key in memory as a six-byte hash across 256 shards and push the values out to disk. One repository dropped from about 800MB in memory to about 40MB of metadata, and the service’s total memory came down from about 2.7GB to about 750MB. The figure that explains the rest came out of the same profile: about 30,000 of more than four million entries are ever held, roughly 1 per cent, at a hit rate above 99.9 per cent.
Pricing a release without knowing what it cost to produce means pricing it off the wrong number, and we say so often. Cloudflare’s on-demand profiling made that case in October from one end, where the cost was simply unmeasured. Uber’s account is narrower and more useful. The cost had been measurable all along. It was being measured against the wrong quantity: the cache was sized from the size of the translation repository, when what governed the bill was how many entries anybody read.
A release whose production cost is unknown is priced from the wrong figure.
The sync reported every task run, and two migrations never ran
Somebody pushed a release while a deny sync window was open. When the window closed, five regional Argo CD applications rolled out the same commit and three of them ran the database migration. The other two reported Synced and Healthy, and their last operation said all tasks had run. New code talked to an old schema for 36 to 44 minutes. Afzal Ansari reproduced the failure five times in six, then read the cause in the Argo CD source: a refresh triggered by a changed object compares the recorded revision against itself, finds nothing new, and heals drift instead of syncing. A partial sync skips every hook and writes no history.
Our position is that work nobody can reconstruct is work nobody can defend. GitHub’s retention change in October was the mild version, where a record expires on a schedule somebody chose. This is the severe one. The record did not go missing. It was written, it said the opposite of what had happened, and the only trace of the truth was a self-heal counter buried in a controller log. Ansari adds that no hook type survives a partial sync, so the usual fix does not reach this.
An output nobody can reconstruct is an output nobody can defend.