The window was smaller than the problem
How we Organise
The cost that grows is the one the benchmark does not sample
Trinath Sai Subhash Reddy Pittala published this on 2 September. A Python program pays for its imports at every process start. That cost is invisible in a steady-state benchmark and dominant for command-line tools, test workers and serverless cold starts. Python 3.15 added explicit lazy imports largely on anecdote, because nobody had measured the ecosystem.
He measured it. The 500 most-downloaded packages, sampled quarterly across five years of versions, under six interpreter versions on two platforms, for 63,431 measurements. Half of the packages import in under 6 ms. The 99th percentile is 354 ms. The first import after installation costs three to twenty-two times more, because the bytecode is being compiled. Importing a package’s submodules costs up to 294 times more than the top-level import a benchmark reports. The median package’s cost grows 1.6 to 2.4 per cent a year. The mean grows 11 to 13 per cent. Newer interpreters import the same code 1.16 times slower on one platform and not on the other, and moving between two point releases swings the cost by 1.34 times. Turning lazy imports on globally makes the statements essentially free and breaks 8 of 414 top packages.
We ask a team to show that consumption per unit of output is falling, or to record the reason it is not. The measurement a team has to hand is a steady-state benchmark. It reports the top-level import. What is growing sits in the submodules and in the tail, where the mean rises four to eight times faster than the median. So a team can report a falling number, honestly, from a measurement that cannot see the part that is rising.
How far this reaches is genuinely open, and we may be reading it too widely. It is one ecosystem, one author and two platforms. Import cost is one component of consumption, and it amortises to nothing in a long-running service. The harness and dataset are available on request rather than published. Nobody can check the reading today. Somebody could fairly hold that cold-start workloads are a small share of what a client spends.
If it stands, we stop telling a client that a falling consumption number is evidence of falling consumption, unless the measurement includes process start.
How we Build
Every leading system found the right code, and put the wrong one first
Aaryan Kapoor and Md Abdullah Al Hafiz Khan submitted ExecRetrieval on 1 September. Embedding-based retrieval sits underneath coding agents. What a benchmark usually asks of it is whether it returns code on the same topic. These authors asked whether it can separate code that works from code that nearly does.
They built a pool that contains the counterfactuals. There are 939 Python tasks. Each has one canonical implementation and up to four buggy variants, every one execution-verified. Every variant is a mechanical mutation making a single targeted edit. Against that pool they ran 23 dense embedding configurations and BM25, invoked through each provider’s own interface, with paired McNemar tests and query-level bootstrap intervals. The top hosted system returns the canonical inside ten results every time, at exec@10 of 1.00. It returns it first only 0.331 of the time. When the four leading systems miss at rank one, what sits there instead is one of that query’s own buggy variants 91.5 to 99.4 per cent of the time. The canonical scores below at least one of its four paired variants in 67 to 78 per cent of queries.
We ask that what an answer needed was retrieved. Retrieval is measured on its own, against a labelled set, and every wrong answer is attributed either to the material or to the reasoning. Separating the two is right, and it is the reason the practice exists. The difficulty is what a labelled set is usually made of. Build it the ordinary way, with one correct implementation among code about other things, and every leading system passes. Put four near-identical wrong versions beside the canonical and the same system leads with a wrong one two times in three. The measurement still passes. The attribution step then reads the material as sound, on the word of a set that could not tell the two apart.
We may be wrong. These are 939 Python tasks, the variants were generated mechanically rather than collected from real defects, and a single-edit mutant is a hard case by construction. An exec@10 of 1.00 says the material was in reach, so a team reranking with an execution oracle recovers it. One answer is that the practice already implies the labelled set should hold the near misses, and the fault is then in how teams build the set rather than in what we ask for.
Should that hold, we stop telling a client that a retrieval measurement taken on its own establishes that the material was there. It establishes that only if the set contains code that is nearly right.
How we Assure
The revocation completed and the earlier work went through anyway
Igor Santos-Grueiro submitted this on 2 September. It names a property he calls effect closure. A grant is closed when the authority already issued under it retains no path to an effect the application rejects, and no further authority can be issued. His checker, EFFECTBOUND, reduces the question to finite control with hidden state. It returns a strategy, a certificate that closure is impossible, or no verdict where the evidence will not decide it. The reduction and the checker’s soundness carry machine-checked proofs.
Across GitHub, Kubernetes, NATS and Kafka, closure fails in three ways. The interface has no control that would do it, clean visible state hides work still running, or the model stops before the last point at which the effect could be prevented. Each has a case. The GitHub tool cannot bind a merge to the commit that was reviewed, and a controlled run confirms it may merge a different one. NATS can report nothing stored and nothing pending. Work already dispatched goes on publishing downstream. In Kafka every broker in a fixed set had applied the revocation, and a request authorised before it could still append. He adds a gate that refuses new use of revoked authority and holds the return until earlier in-flight work finishes. In a test deployment that closes the studied write path without blocking unrelated requests.
We hold that an agent’s authority is scoped to its task and ends with it. What a team reads to decide that is whether the credential was short-lived and whether the revocation completed. On four widely used providers both read true while the authority has not ended, because work authorised a moment earlier can still reach an effect. Authority ends, on this argument, only once issuance has stopped and nothing already issued can reach an effect the application rejects. That moment is later than the one a team currently reads.
The scope could be narrower than we are reading it, and we may have that wrong. It is four providers and one author. The Kafka path studied is synchronous and nontransactional. The checker returns no verdict where evidence is thin, so three named failures are what could be decided rather than a census. A sceptic would hold that this is a property of those provider contracts rather than of agent authority. On that view a team owning its own services can close the frontier itself.
The thing we would have to stop saying is that a completed revocation ends what an agent may do. Every client running agents against a third-party provider has a window after revocation, and nobody has measured it.