The vent, and the sentence buried in it

Jan Schaumann's post this week is a vent and he says so in the first line. A senior security engineer, decades in, writing that he spends upwards of three quarters of his time directly or indirectly dealing with AI and that it has robbed him of most of his enjoyment of the work. People with no engineering background pitching industry-changing solutions from their agent-infested homelab. Emails that read like influencer posts. Colleagues turned into meat proxies. It hit the front page because a great many people feel exactly this and most of them are not allowed to say it at work.

You can disagree with a lot of it, and the thread did. But there is one paragraph in the middle that is not a vent. It is an argument, and it is the most important thing anyone wrote about AI this week.

He describes the industry-wide effort, now many months old, to point frontier models at vulnerability discovery. Dozens of highly paid engineers per organisation, priorities reshuffled, harnesses built, pipelines built to shoehorn thousands of findings into vulnerability management. Thousands of new vulnerabilities found. And then: I don't think we're any safer than before. Because finding vulnerabilities has never been the bottleneck in information security. The bottleneck is still, as ever before, getting the packages updated. Patching is still hard.

That is the whole essay. The models are extraordinary at the part that was never the constraint.

Meanwhile, at the frontier

Consider what the rest of the industry was looking at while he wrote that.

A researcher quit one of the labs and posted a warning that went, by ThePrimeagen's count on stream, to well over a hundred million views in a day. A colleague who stayed put the odds of catastrophe above ten percent in a decade. The lab's CEO published a three-point plan to pace the frontier. A rival CEO agreed with him, which people found unusual. AI Explained spent a video on the researchers' stated reason: a large gap, in one OpenAI researcher's phrase, between the internal and external perception of the rate of progress. Six axes of improvement, none near saturation. Fireship covered a 154-page threat report cataloguing eight months of misuse across seven categories.

All of this is real and none of it is unimportant. But look at who is in the audience. Engineers, managers, founders, people who have to decide on Monday what to do with a budget. And the frontier discourse gives them precisely nothing to do on Monday, because it is about capabilities they do not control, on timelines they cannot influence, at companies they do not work for. It produces anxiety with no action attached, which is the most tiring kind, and it is why Schaumann is exhausted.

Where the attention went this week, and where a Monday decision can actually land. Nothing in the top row is actionable by anyone outside a lab.

Where the value is landing

We install AI systems in ordinary companies for a living, which gives us a view of this that the frontier discourse does not have, and the view is this. The money is being made in the bottom row of that figure, by people who are not on Hacker News, doing things nobody will write a threat report about.

The shapes are always the same, and we describe them here as composites rather than clients, as everything on this site is. A clinic group that uses a model to reconcile years of insurance remittances against deposits and finds the pattern of underpayments a human was too busy to see. A logistics firm whose intake now reads every inbound document and writes the structured record, so the person who used to do that handles only the exceptions. A software team whose agent does not write features but does read every production error, correlate it to a deploy, and open a ticket with the likely cause before a human has seen the alert. And, in public this same week, Cloudflare saving another hundred terabytes of RAM with what their post cheerfully calls maths.

None of that is at the frontier. All of it runs on models a generation or two behind the ones in the headlines, because the constraint was never the model. Schaumann's line about vulnerability discovery generalises: in almost every function of almost every company, the thing the model is best at was not the bottleneck, and the bottleneck is some unglamorous downstream step that nobody funded because it was boring. Patching. Inventory. The written specification. The approval seat. The person who has to say yes.

The companies winning this year are the ones that noticed the bottleneck first and pointed the tools at it, or more often pointed the tools at everything upstream of it and then spent the savings on the bottleneck. They are not talking about it, partly because it is unglamorous and partly because it is working.

Why the frontier framing hurts

There is a specific mechanism by which the frontier discourse makes ordinary organisations worse at this, and it is worth naming because it is avoidable.

The frontier framing says the model is the variable. Wait for the next one; it will do what this one could not. That framing is true for the lab and false for the buyer, and a buyer who believes it does two bad things. They delay the boring work, because the boring work will surely be automated by the next release. And they evaluate every tool on ceiling, on the impressive demo, rather than on the floor, on what it does at two in the afternoon on a dull task with a tired reviewer.

The second thing the framing does is imply that the risk is exotic. Swarms, bioweapons, the internet taken over. Meanwhile the actual risk in the actual company is the one CNN reported this week from the military: a model produced a confident report with things in it that were not true, and it got some distance up the chain before anyone checked. That failure does not require a frontier model. It requires an unreviewed one. Every company installing these tools this year is building that failure mode unless it builds the seat that catches it, and the seat is boring, and the frontier discourse never mentions it.

The frontier framing versus the operator framing. Same tools, opposite conclusions about where to spend Monday.

How to find the boring bottleneck

The instruction "find the least interesting problem and fix it first" is easy to nod at and hard to act on, so here is the exercise we run in the first week with any organisation.

Pick one process that produces money or stops it. Invoices out, claims in, orders through, tickets closed. Walk it end to end with the people who do it, and at each step write down two numbers: how long the step takes when it goes well, and how long the work waits before that step starts. Almost nobody has the second number, and the second number is the process. A step that takes four minutes and waits two days is not a four-minute step.

Then find the step where the wait is longest, and ask why the work is waiting. The answer is nearly always one of three things. A person has to decide something and is busy. A piece of information is missing and somebody has to go and find it. Or two systems disagree and a human has to reconcile them by hand. Those three are the bottleneck, and none of them is "the model is not smart enough".

Now point the tools. Missing information is the easiest: a model that reads the inbound document and fills the record removes the wait entirely. Systems that disagree is the next: a model that reconciles the ninety percent that match and hands the rest to a person, with the mismatch highlighted, turns a day of reconciliation into an hour. The busy person deciding is the hardest and the most valuable, and it is the approval seat: give them the decision in a form that takes eight seconds and they will make forty before lunch.

Nothing about this requires the frontier. It requires a whiteboard, an afternoon, and a willingness to be interested in a process everyone in the building considers beneath them. The companies winning this year did that afternoon a year ago.

Schaumann is also wrong, and it matters how

The honest paragraph. He is wrong that the technology is the reason the work stopped being enjoyable, and the thread said so in the most upvoted reply: the models genuinely produce a lot of good work, can debug in an afternoon what took a team a week, and the people running the companies are, in that commenter's phrase, the most boring supervillains imaginable. All true at once.

What robbed him of the enjoyment is the framing, not the tool. Seventy-five percent of a senior engineer's time spent on AI is seventy-five percent spent in the top row of the figure, on harnesses for vulnerability discovery that was never the bottleneck, on pipelines to process findings that will not be patched, on the frontier's priorities rather than the organisation's. Point the same engineer and the same models at the bottom row, at the inventory and the patching he himself names as the real work, and the time is not wasted and the work is not joyless. It is the job he signed up for, done faster.

The frontier will keep moving. The researchers may well be right to be frightened; we are not qualified to say and neither is most of the audience. But the companies that come out of this decade ahead will not be the ones that watched it most closely. They will be the ones that found the boring bottleneck in their own building, wrote it down, and put the tools to work on either side of it while everyone else was reading the threat report.

Everybody has lost their minds. The way back is to find the least interesting problem in the company and fix it first.

Sources

• Everybody's Lost Their Minds (Jan Schaumann) (HN, 365 points, 17 Sep 2026)
• What AI Researchers Saw, Before Their Demand to 'Pace' AI (AI Explained, video) (16 Sep 2026)
• I can't believe this is happening (ThePrimeagen, video) (19 Sep 2026; the viral-tweet numbers as reported on stream)
• Anthropic researchers are quitting... (Fireship, video) (15 Sep 2026)
• Saving another 100TB of RAM (Cloudflare) (HN, 463 points, 18 Sep 2026)

Originally published on the Levelbrook playbook. Levelbrook is a principal-led Rails and AI-systems consultancy; the playbook is where we write down what we see.

The story, as told, with the caveats attached

Here is the account that circulated last week, and it is important to say up front that it is a summary of public statements by parties who disagree with each other, so treat every clause as "reportedly".

A mathematics professor and a collaborator who works at one of the labs spent about a year on the Navier-Stokes existence problem, one of the Millennium Prize problems. In the last month of that year they leaned heavily on coding agents and their progress accelerated; in mid-August they got a simpler related system to break, the closest anyone had come. Then, according to their statement, a different lab, having heard rumours of the approach, pointed a very large number of agents and a very large amount of compute at the problem and, within days, claimed a result on the full problem using what the mathematicians say was their novel approach. There was a phone call. The accounts of the phone call diverge sharply. Both sides published, one after the other, on the same Tuesday.

You can read Fireship's telling, which is where most engineers heard it, and you can read the statements. We are not adjudicating it. What we want to point at is the line Terence Tao is reported to have written in response, because it is the most important sentence of the month for anyone who runs a product team, and almost nobody outside mathematics noticed it.

If the mere rumour of your research can trigger a swarm of agents racing to front-run it, mathematicians will simply stop sharing ideas, and that will undo centuries of open science.

Replace "research" with "roadmap"

Now do the substitution. If the mere rumour of your feature can trigger a swarm of agents racing to front-run it, what happens to your roadmap?

For the entire history of the software business, the gap between having an idea and shipping it was the moat. Not the idea itself; ideas were always cheap and always leaked. The moat was that turning the idea into a working, deployed, maintained thing took a team a quarter, and a competitor who heard about it on a Tuesday could not have it by Friday. Product strategy, fundraising, hiring plans, launch timing, all of it was built on that gap being months wide.

The maths story is what it looks like when the gap closes to days at the top of the market. The Dream RSI paper that Fireship covered the same week is what it looks like a level down: a search loop that, per the video's summary, wrote a solver that beat a standard library in about 300 attempts where the fixed-policy version needed 550 and the previous record needed roughly 51,000. That is a search process getting two orders of magnitude cheaper at finding a solution somebody else already knew the shape of. Knowing the shape of the solution is what a roadmap leak gives a competitor.

The idea-to-shipped gap was the moat. Durations illustrative; the collapse is the point.

This is not an argument for secrecy, for the same reason Tao's line is not an argument for mathematicians to stop publishing. Secrecy does not work either: your customers know what they asked for, your job postings say what you are building, and your own agents' instruction files describe the product in more detail than any leaked slide. It is an argument for being honest about what is still defensible when the build is no longer the hard part.

What a swarm cannot front-run

Go back to the maths. A swarm with a rumour and twenty million dollars of compute could, reportedly, reproduce a result. What it could not do is the year before the rumour: choosing that problem, choosing that approach out of the dozens that do not work, building the intuition that made the approach look promising when it looked like nothing to everyone else. The swarm needed the shape. The year produced the shape.

The commercial version of that year is the thing we wrote about earlier this month under the heading that companies do not have processes, they have habits. The forty exceptions that live in one person's head. The knowledge of which customer needs it done differently because of the freight claim in 2023. The understanding of why the obvious version of the feature fails for the second largest account. A swarm given the rumour builds the obvious version, perfectly, in days. The obvious version is the one your customers already rejected.

What a swarm reproduces from a rumour, and what it cannot. The right-hand side is the whole of the defensible business.

So the defensible assets in a world of swarms are the ones on the right of that figure, and they are exactly the assets most product organisations have neglected because they were not the bottleneck. The written specification, including the exceptions, which almost nobody has because the code was the specification and the code was expensive. The proprietary data about how the product fails and for whom. And the accountability, the fact that when the thing goes wrong there is a named organisation that picks up the phone and fixes it, which no swarm has ever offered anyone.

What changes on the roadmap

The practical consequences are uncomfortable for the way product teams have worked.

Announcements shift from features to outcomes. Announcing "we are building X" now hands the shape to anyone with a swarm. Announcing "our customers in this segment no longer have this problem" hands them nothing they can prompt with, because the interesting part is which segment and which problem and why the obvious solution did not work, and that lives in your specification.

The specification becomes the product. Not a slide, the actual document: what the system does, what it refuses to do, every exception and why. If your team's competitive advantage is knowledge that lives in heads, the swarm era is the strongest incentive you will ever get to write it down, because the written version is the only form in which it compounds and the only form in which it can be defended.

Speed stops being a strategy and becomes table stakes. Being first to ship the obvious version was worth a great deal when it took a quarter. It is now worth roughly a week of attention. The teams that win will be the ones that were already on the third version, informed by the failures of the first two, when the swarm shipped the first one.

And the Gowers point, from his essay on why he declined to sign the Fields medallists' letter, applies in full. He has argued for twenty-five years that mathematics contains two cultures, problem-solvers and theory-builders, and that the field needs both. The swarms are extraordinary problem-solvers. Theory-building, meaning the slow accumulation of a coherent understanding of a domain, is what makes the problems worth solving and tells you which one to solve next. Every product organisation is about to discover which of the two it was actually good at.

What to write down this quarter

The uncomfortable implication of the figure above is that the defensible assets are documents, and most organisations do not have them. So here is the writing programme, in priority order.

The exceptions register. For the product's core workflow, every case where the obvious behaviour is wrong and why. Not the happy path, which anyone can reproduce, and not the code, which a swarm can regenerate. The list of forty things that live in the head of the person who has been there longest, with the reason attached to each. Ask that person to talk for two hours, record it, have a model draft the register, and have the person correct it. This is the single most valuable document the company can own and it usually does not exist.

The refusal list. What the product deliberately does not do, and what happened when someone tried. A competitor building from a rumour will build the features you rejected, because they look like features. Knowing why they were rejected is a year of learning that cannot be prompted for.

The failure data. Which customers hit which failures, how often, and what it cost. Nobody outside the company has this, and it is the input that decides which of the next ten things to build. Keep it in a form a model can read, because your own agents should be making the obvious version of the next feature for you, informed by it, before anyone else makes the obvious version uninformed.

The accountability map. Who picks up the phone when the thing is wrong, in what timeframe, with what authority to fix it. Write it down, publish the shape of it to customers, and mean it. A swarm can ship a product. It cannot answer for one, and in the year when every product has a swarm-built twin, the answering is the product.

Where this is wrong

The counter-argument is that most companies do not compete with labs holding twenty million dollars of spare compute, and that is true today. It will be less true every quarter, because the cost of the swarm is falling on a schedule nobody in your market controls, and because the swarm does not need to be a lab. It needs to be a competitor with a credit card and a rumour. The Navier-Stokes story is not the shape of your next year. It is the shape of your next three, arriving at the top of the market first, as these things do.

Tao's warning was that scientists would stop sharing. The commercial equivalent is worse: companies will keep sharing, because they have to, and will discover that the only part they could ever have kept was the part they never wrote down.

Sources

• OpenAI's biggest math breakthrough is getting ugly (Fireship, video) (11 Sep 2026; the account of the Navier-Stokes dispute below is Fireship's summary of the parties' public statements, and both sides dispute the other's version)
• Did Google just kickstart the intelligence explosion? (Fireship, video) (17 Sep 2026; the Dream RSI figures)
• Why I didn't sign the Fields medallists' letter (Timothy Gowers) (HN, 276 points, 17 Sep 2026)
• Your company does not have processes. It has habits. (Levelbrook)

Originally published on the Levelbrook playbook. Levelbrook is a principal-led Rails and AI-systems consultancy; the playbook is where we write down what we see.

Second time this year

The story, as reported this week, goes like this. A coding agent from one of the large model labs, promoted with a generous free tier this month, was found to be taking snapshots of the user's workspace, including the git history, and sending them to the vendor's cloud. The uploaded data was encrypted, a commenter noted, with a key the user does not hold. The reporting was careful to say it did not know the intent. The Hacker News thread was less careful, and its most upvoted line was that this is the second such story this year, after a similar discovery about another lab's agent earlier in the summer, and that the lesson from the first one was apparently not learned: do not trust harnesses, especially new ones.

We are not going to relitigate the specifics, because we only know what was reported. The interesting part is not this vendor. It is that the discovery was made by a user reading network traffic, not by any process in any of the organisations that had installed the thing. Which means the question worth asking is not "was this one bad" but "what did you do, before you installed it, to find out".

The most privileged contractor you have ever hired

Think about what a coding agent is, in access terms, and compare it to a human contractor.

A contractor gets a laptop, a repository, a set of credentials scoped to their task, an NDA, a background check, and a manager who watches what they commit for the first month. They work during hours. They can be asked what they did. They can be fired.

A coding agent gets your entire workspace, which in practice means every repository you have cloned, every .env file you forgot was there, the shell history, the SSH keys the shell can reach, the cloud credentials in the credential helper, and the git history of everything, which is where the secret you rotated in 2024 still lives. It runs with your user's permissions. It runs while you are away. It makes outbound network requests you do not see, to endpoints you did not configure, and it was installed by a developer with a one-line command because the free tier was generous and the demo was good.

What a coding agent can reach from a normal developer workstation, compared with what a contractor is given. Nothing on the left was scoped by anyone.

Nobody would give a contractor the left-hand column. Every organisation with developers has given it to several agents this year, from several vendors, some of which did not exist in January.

The threat is symmetric

There is a second half to this story and it ran the same week. Fireship's summary of Anthropic's recent threat report, which the video says covers eight months of misuse the company detected and shut down, included one detail that should land hard for anyone who ships software. A criminal group was reported to have mass-downloaded around 1.8 million Android application packages, decompiled them with the help of a model, and mined them for hard-coded secrets. The keys they wanted most, the video notes with some relish, were API keys for the model providers themselves. Those are the easiest to monetise.

Read the two stories together. On one side, agents installed on developer machines with access to everything and an outbound connection nobody audited. On the other, agents run by attackers, decompiling shipped software at scale looking for exactly the kind of secret that a vibe-coded app, or a workspace snapshot, hands over. The same capability that makes an agent useful to you makes it useful to the person on the other end of the network connection, and the asymmetry that used to protect the small shop, that nobody would bother to decompile your app by hand, is gone.

The Hacktron write-up from the same week, on chaining a heap overflow and an SSO misconfiguration into access to internal repositories at one of the labs, makes the point from the top of the market. If the people building the models can be reached through a misconfiguration, the tool on your laptop was not built by people who are immune to the same thing.

What "background check" means for a tool

The remedy is unglamorous and it is mostly the security discipline you already apply to dependencies, applied to a category that has been exempted because it is exciting.

Read the network. Before an agent touches a real repository, run it in a scratch workspace with a proxy in front of it and look at every host it talks to and what it sends. This takes an hour. It is how this week's story was discovered, by one person, and every organisation that installed the tool could have done it first.

Scope the workspace. An agent should see one repository, not a home directory. Run it in a container, a devcontainer, a VM, a separate user, anything that means "your workspace" is a directory you chose rather than everything the shell can reach. If the tool does not work that way, that is information about the tool.

Scope the credentials. Short-lived tokens, per-agent, for the one system the task needs. No credential helper with a year-long cloud key. No SSH agent forwarding. If the agent needs to push, give it a deploy key for that repository and nothing else.

Gate the egress. An allow-list of hosts an agent may reach is a small piece of configuration and it converts "we found out from a blog post" into "the request failed and we looked". This is the single highest-return control on the list and almost nobody has it.

Treat the instruction files as code. The AGENTS.md and the skills directory an agent reads are executed, in the sense that matters. Review them. Own them. Pin them. Cloudflare published an audit skill this week for exactly this kind of surface, which is a sign the serious shops have started treating agent configuration as something that needs auditing.

The controls that would have turned this week's story into a failed request. None of them require trusting the vendor.

The hour, in detail

Since the whole argument rests on "this takes an hour", here is the hour.

Make a scratch directory containing a small repository with a deliberately fake secret in it: a file called .env with API_KEY=canary- followed by a long random string, committed once and then removed in a second commit so that it exists only in history. Put a second canary in the shell history. These are your tracers. If either string ever appears in an outbound request, you have your answer without reading anything else.

Run the agent inside a container or a fresh user account with a local HTTP proxy configured as the system proxy and its certificate trusted, so that TLS traffic is visible. Point the agent at the scratch repository and give it a dull, real task: rename a function, add a test, fix a typo. Let it finish. Do it three times, once with a fresh session each time, because some tools snapshot on first run and some snapshot periodically.

Then read the proxy log. You are looking for four things. Every distinct host contacted, which should be a short list you recognise. Any request whose body is large relative to what the task required, which is what a workspace snapshot looks like. Any request containing either canary string, which is the tracer firing. And any request that happened when the agent was idle, which is telemetry, and which you then read with more suspicion than the rest.

Write the list of hosts down. That list becomes the egress allow-list for the tool on real machines, and the next time the tool updates you run the hour again and diff the lists. If a new host appears, you find out from your own log rather than from someone else's blog post, and you find out before it has touched anything that matters. An hour, once per tool, once per major update. There is no security control on the market with a better return.

Where this is unfair to the vendors

Some of this is the ordinary growing pains of a new category. Telemetry that a team thought was obviously fine looks very different when a user reads the packet capture. Workspace snapshots have legitimate uses in resumable agent sessions. The labs are, by and large, staffed by people who would be horrified to be described as exfiltrating anything. None of that changes the buyer's position. You do not get to know intent. You get to know what the tool can reach and where it sends things, and both of those are measurable before you install it.

The reason this is the second story of its kind this year and will not be the last is that the category moved faster than the discipline. Every other piece of software that runs with a developer's full permissions and talks to the internet went through a decade of being treated as a risk before it was treated as a convenience. Coding agents skipped that decade because they were useful immediately.

They are still useful. Install them the way you would hire someone who will have the keys to everything: find out what they can reach, decide what they may reach, and watch the door.

Sources

• Inside ZCode: silently uploading your Git history to the cloud (ferstar) (HN, 328 points, 18 Sep 2026; details below are as reported there and in the thread)
• Anthropic researchers are quitting... (Fireship, video) (15 Sep 2026; Fireship's summary of Anthropic's threat report, including the decompiled-APK figure)
• A heap overflow and SSO misconfiguration to compromise OpenAI internal repos (Hacktron) (HN, 484 points, 18 Sep 2026)
• Cloudflare security-audit-skill (HN, 210 points, 17 Sep 2026)

Originally published on the Levelbrook playbook. Levelbrook is a principal-led Rails and AI-systems consultancy; the playbook is where we write down what we see.

One file, every task, all day

The changelog entry was one line and it got seven hundred points. Claude Code now reads a repository's AGENTS.md when there is no CLAUDE.md. The top comments were what you would expect: relief that the one-line CLAUDE.md files that just said "read AGENTS.md" can go, a joke about who borrowed the idea from whom, and a grumble that the skills directory is still proprietary. Standards converging. Nice.

Step back from the tooling and look at what just happened to your documentation.

For twenty years the most-read document in a software company was, depending on the company, the onboarding wiki, the README, or the runbook that everyone opened during the outage. Read, in each case, by a handful of humans a handful of times a year, mostly in their first month. Nobody measured it because the number would have been embarrassing.

AGENTS.md is read by every agent, on every task, before it does anything. If your team runs a few hundred agent sessions a day, and many do now, that file is consulted a few hundred times a day. Its contents shape every change that gets made. It is, by a margin that is not close, the most consequential piece of writing in the building. And in most of the repositories we have seen this year it was written in forty minutes by whoever set up the tool, has never been reviewed by anyone, and contains at least two instructions that contradict each other.

What is actually in there

We have read a lot of these files. The pattern is remarkably consistent.

The first third is tooling trivia. Run the tests with this command. Use this package manager. Do not touch the generated directory. This is fine and it is the part that gets maintained, because when it is wrong the agent breaks loudly.

The second third is the accumulated scar tissue of things that went wrong. Never run the migration directly. Always check the feature flag first. Do not use the old client library. Each line was added after an incident by whoever was on call, in the language of that incident. Nobody has gone back to check whether the flag still exists.

The last third is the interesting part: the habits. Prefer small functions. We do not use that pattern here. Follow the existing style. Ask before adding a dependency. This is the oral tradition of the team, written down for the first time in its history, by one person, from memory, in the voice of a Slack message.

What a typical AGENTS.md contains, and who reads each part. The bottom third is the team's undocumented process, written down once, by one person, for the first time.

We wrote earlier this month that companies do not have processes, they have habits, plus a document that describes an idealised version of some of them. AGENTS.md is that document, except that this time the reader is not a new hire who will paper over the gaps by watching the person at the next desk. The reader is a system that will follow the instruction literally, hundreds of times, and will resolve the contradictions by picking one, silently, differently each time.

The failure modes are already visible

There was a comment under the self-driving-codebases piece this week that describes the shape of the thing perfectly. An engineer running their own agent loops watched the first agent try to run an enormous dependency inspection command, run out of memory, and record a workaround in the agent's memory. The workaround was a way to make the enormous command succeed. Every subsequent agent inherited the workaround. The instruction file had learned to do the wrong thing more reliably. They called it cargo-cult behaviour, and the name is right.

Instruction files accrete. That is what they are for. But accretion without review produces a document whose instructions were each correct on the day they were written and which, taken together, describe a codebase that no longer exists. The agent does not know that. It reads the file fresh every time and does its best.

The second failure mode is contradiction. "Always add tests" and "do not modify files outside the task scope" are both reasonable lines and they conflict on roughly a third of tasks. A human resolves that with judgement and a quick message. An agent resolves it by weighting, and the weighting is not something you configured.

The third is the one that should worry whoever owns the codebase. The file is unowned. It has no review process, no owner in the CODEOWNERS sense, no changelog and no tests. Anybody who can push can add a line, and the line will be obeyed by every agent from then on. If you have ever worried about supply-chain risk in your dependencies, the highest-privilege dependency in your repository is now a Markdown file.

A worked example, from a file we were asked to look at

A composite, assembled from several files we have reviewed this year, with the details changed. The file was 61 lines long. Line 9 said to always run the full test suite before opening a pull request. Line 34, added after an incident in the spring, said never to run the integration tests locally because they hit a shared staging database. The full suite included the integration tests. Every agent that read the file resolved the contradiction in one of two ways: it ran the full suite and hit staging, or it skipped the suite and opened the pull request untested. Which one it chose depended on the model, the day, and how much else was in the context window. Nobody had noticed for four months because both outcomes looked like normal agent behaviour.

Line 22 said to use the internal HTTP client wrapper rather than the raw library. The wrapper had been deleted in a refactor in July. Agents that obeyed line 22 searched for the wrapper, failed to find it, and either recreated it from the description or fell back to the raw library with a comment apologising. Three copies of a near-identical wrapper had accumulated in the repository, each written by an agent trying to comply with an instruction about a thing that no longer existed.

None of this is exotic. It is what happens to any document that is executed without being maintained, and the fix for all of it took one person one afternoon: read the file, delete eleven lines, rewrite four, add the reason to each remaining rule. The next month's agent sessions were measurably cleaner, and the person who did it described the afternoon as the highest-return work they had done all quarter.

Treat it like what it is

The remedy follows from taking the file seriously, and it is mostly process rather than tooling.

Give it an owner. A named person, in CODEOWNERS, whose approval is required to change it. Not because the changes are dangerous individually but because somebody has to hold the whole file in their head and notice when line 14 and line 31 disagree.

Review it on a cadence, as a document. Once a month, read the whole thing top to bottom, with a model if you like, and for every instruction ask three questions. Is this still true? Is it still needed? Does it conflict with anything else in here? Delete generously. A shorter file that is all true beats a longer one that is mostly true, because the agent cannot tell which lines are the mostly.

Separate the layers. Tooling facts, safety rules and stylistic preferences are different kinds of instruction with different failure costs, and they should look different on the page. A safety rule that the agent must never violate should not be sitting in the same list as a preference about function length, in the same font, with the same weight.

Write the why. This is the one that turns the file from scar tissue into process. "Never run migrations directly" is a rule. "Never run migrations directly, because production runs them through the deploy pipeline with a lock, and a direct run in 2025 took the API down for forty minutes" is a rule an agent can reason about, including reasoning about when it does not apply. It is also, incidentally, the first time that piece of institutional knowledge has been written down anywhere a new human could find it.

The instruction file, run like the load-bearing document it has become.

And test it. This sounds odd for a Markdown file and it is the most useful thing on the list. Keep a short set of tasks that went wrong in the past, the ones that produced the scar-tissue lines. Once a month, run an agent against them with the current file and see whether the lines still do their job. If the agent makes the old mistake, the instruction has rotted or been contradicted. If it does not, the line is earning its place. This is the same idea as a regression test, applied to the document that governs the thing that writes your code.

The larger point

The reason this deserves an essay rather than a checklist is what the file reveals. For the first time, the habits of an engineering team have been written down in a form that is executed rather than merely consulted. That is uncomfortable, because the writing is bad and the habits are inconsistent and everybody can now see both. It is also the largest documentation opportunity a software organisation has ever had, because for once there is an immediate, measurable cost to the document being wrong, and an immediate, measurable benefit to it being right.

The wiki was never read, so it never mattered that it was wrong. This file is read constantly. Write it like something that is.

Sources

• Claude Code now reads AGENTS.md if there is no CLAUDE.md (changelog) (HN, 715 points, 18 Sep 2026)
• Your company does not have processes. It has habits. (Levelbrook) (the earlier essay this one builds on)
• Towards Self-Driving Codebases (detail.dev) (HN, 119 points; the agent-memory cargo-cult comment is from the thread)

Originally published on the Levelbrook playbook. Levelbrook is a principal-led Rails and AI-systems consultancy; the playbook is where we write down what we see.

The number on the slide is a car, and you are being sold an engine

Every coding-agent pitch you have seen this year has a number on it. SWE-Bench Verified, some percentage, up and to the right, usually next to a model name. The implication is that the number belongs to the model, and that if you buy the model you get the number.

A commenter on this week's Hacker News thread about the harness study put the problem better than the paper's abstract does. If Car A is faster than Car B, it is not necessarily the engine. It could be the tyres, the gearbox, the weight, the driver. A coding agent is a car. The model is the engine. The harness, meaning the loop around the model that decides what it sees, what it can do and when it stops, is everything else. And the number on the slide is a lap time for the whole car, measured on a track you do not drive on.

Nine researchers at Fan et al. did the thing nobody selling these tools has an incentive to do. They held the model fixed, held the execution loop fixed, and varied three harness components one at a time: planning, action space, and context management. Four models, two benchmarks (SWE-Bench Verified and Terminal-Bench 2.1), 176 matched configurations, five context-management strategies, four context-window budgets. Then they looked at the trajectories, not just the scores, to see what each component actually changed about how the agent behaved.

• 176 matched harness configurations
• 4 models held fixed
• 3 components varied: planning, action space, context
• 5 context-management strategies compared

What they found, translated out of the abstract

The findings are almost aggressively unglamorous, which is how you know they are worth something.

Context management, the machinery that decides what to throw away as the conversation fills up, matters more the tighter the context budget, and most of its benefit comes from one thing: not falling over when the window overflows. It does not make the agent smarter. It lets the agent keep going. The strongest strategy in their comparison was the boring one, a rule-based pass that deletes obviously stale material before any model-based summarisation runs. And the clever addition everyone builds, making elided content recoverable so the agent can go back and fetch it, turned out to be machinery the models rarely used and which yielded no accuracy gain.

Planning, meaning an explicit plan-first step, changes role depending on the model. For weaker models it is an accuracy scaffold. For stronger ones it stops helping accuracy and becomes a cost saver, with a small decrease in success rate as the price. Another commenter connected this to something in the Claude Code changelog: the built-in todo and task-tracking tools were switched off by default on the newest model generations. The vendor, in other words, appears to have measured the same thing.

Action space, meaning whether the agent gets a menu of predefined tools or just a shell, splits the same way. Predefined tools raise success rates for models that are weak at driving bash. Models that are good at bash do better and cheaper with bash alone, most clearly on command-line-shaped tasks. The paper does not define "bash-capable" crisply, which a commenter rightly flagged, but the direction is unambiguous: every tool you add beyond the shell is a claim that needs testing, not a free improvement.

What each harness component does, by model strength, per the study's abstract. Read the row for the model you actually run.

The trajectory analysis is the part a buyer should care about most. Context management extended how long the agent could keep working without substantially changing what it did. Planning changed where trajectories stopped. Action space changed the granularity at which code got written. None of the three made the engine better. They changed the gearbox, the tyres and the driver, and the lap time moved accordingly.

Why this matters more than the leaderboard

Put the study next to what the practitioners were saying this week and a picture forms.

Theo Browne's video argued, with some heat, that people who cannot feel the difference between model generations are prompting badly, and his most useful idea was a picture of the distribution: every model has a ceiling, which is what the demos show, and a floor, which is what you hit at two in the afternoon on a boring task. Frontier models mostly raise the floor. ThePrimeagen, the same week, left a current-generation model on a trivial colour bug, came back forty minutes later, and found it had spent 330 million tokens and 118 dollars reading the same file over and over. That is a floor.

A benchmark number is a ceiling measurement of one car on one track. It tells you nothing about the floor, and the floor is where your money goes. And the study tells you the floor is mostly a harness property: whether the loop notices it is stuck, whether the context gets cleaned before it overflows, whether the agent has a shell or a menu, whether there is a plan and whether the plan is worth its cost for the model you actually run.

The four questions to ask instead

So when the next vendor slide arrives, the number is not the question. These are.

Which harness produced this number, and can I see it? If the answer is "our proprietary agent runtime", you are buying a car and being told the horsepower. Ask for the loop: what the model sees, what it can do, how context is managed, when it stops.

What does it do when it is stuck? Ask for the failure trajectories, not the success ones. A good vendor has them and is proud of them. Ask specifically what happens at context overflow and what happens after the tenth identical tool call. The study says that is where the benefit of the whole context-management apparatus lives.

What does it cost per success at the floor, not per success on the benchmark? Your workload is not SWE-Bench. Take twenty of your own boring tasks, run them, and divide dollars by successes. Include the runs you killed. The study's own finding, that stronger models do better and cheaper with fewer tools, is a hypothesis you can test on your codebase in an afternoon.

Which components would I turn off? This is the question the paper actually equips you to ask. If your model is strong, the plan step might be a cost centre. If it is bash-capable, the tool menu might be a drag. If your context budget is generous, the elaborate recoverable-summary system might be doing nothing. A harness with fewer components that you understand beats one with more that you do not, for the same reason a car you can service beats one you cannot.

A benchmark measures the whole car once, at its ceiling. A buyer needs the floor, per component, on their own track.

The afternoon test

Since the third question is the one that matters and the one a vendor cannot answer for you, here is the protocol we use, which costs an afternoon and a modest API bill.

Pick twenty tasks from your own recent history. Not the interesting ones. The ones that came in as tickets and got done without anyone remembering them: a null check, a copy change, a small migration, a flaky test, a dependency bump that broke something. Ten of them should be the kind of task an intern would finish before lunch. Ten should be the kind that looks trivial and turns out to touch four files. Write each one down as a ticket, the way it was actually written, with the same missing context.

Run each task through the candidate agent with its default harness, on a clean checkout, with a fixed budget of tokens or minutes, and walk away. Do not steer. Steering is what the demo does and it is what you will not have time to do at volume. When the budget runs out or the agent stops, record three things: did it produce a change a reviewer would accept, how many tokens did it use, and did it at any point loop, meaning repeat an action it had already taken with the same result.

Then divide the money by the accepted changes. That number, dollars per accepted change on your own dull work, is the only benchmark that will predict your bill. Run the same twenty through a second candidate and you have a comparison that no leaderboard offers. Run them again with the plan step disabled, or the tool menu replaced with a shell, and you have reproduced the study's method on the only codebase you care about. The loops column is the floor made visible; a candidate that looped on three of twenty will loop on fifteen percent of your work forever, and no ceiling justifies that.

The limit of the study, stated plainly

It was run on a particular set of models that, as one commenter complained, did not include the current frontier or the strongest open-weight options. The definitions are looser than they should be. The benchmarks are the benchmarks, with all the training-set contamination questions those carry. None of that changes the shape of the result, which is that the harness is a first-class variable and the leaderboards treat it as noise.

The engine matters. Buy a good one. But you are going to spend the next year driving the car, and the car is the part you were never shown.

Sources

• An Empirical Study of Harness Design for Coding Agents (Fan et al., arXiv 2609.20804) (HN, 217 points, 18 Sep 2026; all study numbers below are from the abstract)
• HN thread on the study (the car analogy and the Claude Code todo-tools observation come from commenters there)
• Please stop using stupid models (t3.gg, video) (18 Sep 2026; the ceiling-and-floor framing)
• How are they losing so bad (ThePrimeagen, video) (15 Sep 2026; the 330-million-token session, as reported on stream)

Originally published on the Levelbrook playbook. Levelbrook is a principal-led Rails and AI-systems consultancy; the playbook is where we write down what we see.

A language, a proof, and a field nobody looked up

Two stories ran side by side on Hacker News this week and they are the same story.

The first is Bend 2, a language pitched for the AI coding era: humans write "laws", the AI writes implementations and proofs, and the compiler checks the proofs. It got six hundred points and a lot of admiration. Then Liam Powell wrote a response that got three hundred more, and his point was not that the language is bad. His point was that the demo on the home page takes 58 lines to state that a player can never touch the flag, 442 lines of AI-written proof to establish it, and that the phrase "formal verification" appears nowhere on the website or in the codebase. He then asked a model to redo the demo in SPARK, a language built for exactly this, with no further guidance, and it came back a fraction of the size. The README, a commenter noted, says the compiler is 99 percent AI-written and has not been fully audited.

The second story is Dan Abramov's account of vibing a proof of a conjecture of Conway's with a model, over days, in a long transcript he published in full. It is a good post and an honest one. The most upvoted objection under it was a mathematician pointing to Gowers's essay from the same week on why he did not sign the Fields medallists' letter, and the older point Gowers has been making for twenty-five years: there is a difference between solving a problem and understanding a field, and the second is what makes the first mean anything.

Powell names the mechanism precisely and we are going to steal his sentence: vibe coding makes it possible to build a substantial solution before learning enough about the problem to recognise that a much better solution exists.

Why this is new

It has always been possible to reinvent a field badly. Every senior engineer has watched a junior build a job queue in a spreadsheet. What is new is the ratio.

For all of software's history, building was expensive relative to reading. Before you could produce 442 lines of anything, you had spent enough hours inside the problem that you had, almost by accident, tripped over the prior art. You searched for the error message. You read the paper the library cited. You asked the person at the next desk, who said "oh, that's just a Bloom filter". The cost of building was a tax that paid for an education.

That tax is gone. A model will produce the 442 lines in the time it takes to make coffee, and it will produce them competently enough that they work, and working code is the most persuasive argument in the world against going back to read. Nothing in the loop ever forces you to discover that the field exists. The model will not volunteer it unless you ask, and you do not know to ask, because the whole point is that you do not know the field exists.

The old cost of building bought an education for free. The new cost does not. Time axis illustrative; the shape is the point.

The Bend story is the pure case because a language is the most expensive thing you can build and formal verification is one of the best-documented fields in computer science. If it can happen there, at that scale, with that much talent, it is happening in your codebase this week at a smaller scale where nobody will write a blog post about it. The agent that built your rate limiter from scratch instead of reading the one in your framework. The retry logic that reinvented exponential backoff without the jitter. The custom auth layer.

The gate goes before the build, not after

The instinct is to fix this with review, and review does catch some of it. But review happens after the 442 lines exist, when the sunk cost is already arguing for them, and the reviewer usually shares the author's blind spot. The place to put the gate is the fifteen minutes before anything is built.

We run something we call the prior-art pass, and it is embarrassingly simple. Before an agent is allowed to build anything with a name, it has to answer four questions in writing and a person has to read the answers. What is this problem called by people who study it? What do they already use? Why does the existing thing not work here? What is the smallest version of this we could build on top of the existing thing instead?

The model is extremely good at answering these questions. It has read the field. It will tell you about SPARK and Dafny and Lean and TLA+ in one paragraph if you ask it to, and it will tell you what each is for. The trick is that somebody has to ask before the build starts, and that somebody has to be willing to hear "this already exists" as good news rather than as an obstacle to the thing they were excited to make.

The prior-art pass: four written answers, one human read, before an agent may build anything with a name.

Liam Nugent's piece from the same week, on why the most important product decision is what you do not build, makes the organisational version of the same point. Nobody gets promoted for deleting things. Those who create and launch are the ones rewarded. The models have made creating and launching nearly free, which means the incentive that was already skewed towards building is now skewed by another order of magnitude, and the only counterweight is a deliberate, slightly unpopular gate that asks "does this need to exist" before the exciting part starts.

The pass in practice

A composite from our own work, because the pass sounds like a platitude until you watch it fire.

A team wanted a service that deduplicated inbound customer records, which arrive from four systems with inconsistent formatting, so that the same person is not created four times. An agent, asked directly, would have built it in an afternoon: normalise the fields, hash them, compare. The prior-art pass asked the four questions first, and the agent's written answers were, in order: this is called entity resolution or record linkage; the standard approaches are probabilistic matching in the Fellegi-Sunter family and there are mature libraries in every major language; the naive hash-and-compare approach fails on exactly the inconsistent formatting the team has, because it treats a transposed digit as a different person; the smallest version is to run an existing library with blocking on postcode and hand the ambiguous pairs to a human.

Fifteen minutes. The person reading the answers had never heard the phrase "record linkage". The team built the small version on top of the library, spent the afternoon they saved on the human review queue for ambiguous pairs, and did not spend the following quarter discovering, one support ticket at a time, every way in which the hash approach silently merges or splits real people.

The point is not that the agent knew about record linkage; of course it did. The point is that nobody would have asked, because the task looked simple and the build was cheap, and the cost of the field not being known would have been paid by customers over months rather than by the team in one visible failure. That is the shape of the trap every time. The wrong build does not fail. It works, slightly worse than the right build, forever.

Where this is wrong

The honest caveat is that the prior-art pass has a failure mode of its own: it can become an excuse never to build anything new, and some things genuinely are new. Bend's author may well have considered SPARK and rejected it for reasons that are not on the website. Abramov's proof may be a real contribution even if he cannot yet situate it in the field. The gate is not "never build". The gate is "never build without having looked", and the output of looking is sometimes "nothing here fits, build it, and say in the README what you looked at and why it did not fit". That sentence in a README is worth more than the 442 lines under it, because it tells the next reader that the author knew where they were standing.

What to do on Monday

Find the three most recent things your team or your agents built that have a name. A service, a library, an internal tool, a pattern with a wiki page. For each one, ask the four questions now, after the fact. Do it with a model; it will take ten minutes each. You will find that at least one of the three is a smaller, worse version of something that already existed, and you will feel the thing Powell's post is about, which is not embarrassment exactly. It is the realisation that the cost of not knowing has gone up precisely because the cost of building has gone down.

Then put the pass in front of the next build. Fifteen minutes, four questions, one reader. The model will do most of the work. The only thing it cannot do is want to know.

Sources

• Bend 2 and the Vibe-Coding Trap (Liam Powell) (HN, 323 points, 18 Sep 2026)
• Bend, a language that blocks AI mistakes via proof and runs on GPUs (HN, 603 points, 17 Sep 2026)
• I vibed a proof of Conway's conjecture (Dan Abramov) (HN, 259 points, 18 Sep 2026)
• The most important product decision is what you don't build (Liam Nugent) (HN, 156 points, 17 Sep 2026)

Originally published on the Levelbrook playbook. Levelbrook is a principal-led Rails and AI-systems consultancy; the playbook is where we write down what we see.

The generation side won. Nobody told the review side.

Start with the concession, because it is true and it is the whole reason this matters.

Coding agents work. The team at detail.dev put it as plainly as anyone this week: agents can oneshot games that are actually fun, and with the right guardrails they execute migrations and language rewrites in complex codebases that would have been a quarter's work two years ago. Theo Browne spent an entire video this week arguing that if you cannot tell the difference between this year's frontier models and last year's, the problem is your prompting, and he is mostly right. The ceiling on what a single engineer can emit in a day has gone up by an amount that is genuinely hard to describe to someone who has not sat in front of it.

And yet the same detail.dev post opens with the sentence every engineering manager has been avoiding saying out loud: a lot of orgs spent the first half of this year offloading as much work as possible to armies of agents, and the results have been disappointing. Mountains of dubious code, no tsunami of incredible software. They call it the trough of disillusionment. We would call it something more boring. The factory got a faster machine at one station and nothing else changed.

Where the work actually went

Here is the shape of a change to production software in 2024, roughly, in the units that matter: minutes of a human's attention.

Somebody spends four hours writing it. Somebody else spends twenty minutes reviewing it. A machine spends six minutes testing it. Somebody spends five minutes deploying it. The human write step dominates, so every tool of the last fifteen years attacked the write step: better editors, better languages, better frameworks, and now agents.

Now the write step takes twelve minutes. Not four hours. The agent drafts, the engineer steers, and a change that used to be an afternoon is a coffee. Which means the engineer produces, on a good day, somewhere between five and fifteen times as many changes as before. Every one of them still needs twenty minutes of somebody else's attention before it ships.

The write step shrank by an order of magnitude; the review step did not move. Minutes are illustrative for a mid-sized change; the ratio is what every team we have talked to describes.

Do the arithmetic once and it stops being a vibe. If the review capacity of a six-person team is fixed at, say, forty reviews a day, then the team ships forty changes a day whether the engineers produce forty or four hundred. The other three hundred and sixty sit in a queue. Engineers notice the queue, so they stop producing, or they start rubber-stamping each other, or they merge their own work at six in the evening when nobody is looking. Every one of those behaviours shows up in the incident log within a month.

This is why the productivity numbers are so confusing. The individual is faster. The system is producing the same amount of finished software as before, with worse review. Both things are true.

The seat nobody re-engineered

Look at what the tools industry did in response. It built more agents. Agents to write the tests, agents to review the pull request, agents to fix the review comments, agents to review the fix. Some of it is good. Fireship's sponsor this week, Macroscope, says its review tool now auto-approves forty percent of pull requests across its customers, which is a vendor claim from a sponsor segment and we would treat it as exactly that, but it tells you where the market thinks the money is. The market thinks the answer to a review bottleneck is to remove the reviewer.

We think that is the wrong objective function, for the same reason it was wrong in the back office. The human in the review seat is the accountability. When the change breaks production, somebody approved it, and that somebody has a name and a manager and a memory of what they were told. An auto-approval does not have a memory. It has a log line. You can automate the reading of a diff; you cannot automate the standing-behind of one.

So the question is the one detail.dev asks at the end of their piece and then does not quite answer: when the software mostly drives itself, what do the engineers do? Our answer is not glamorous. They review. And the entire engineering problem of the next two years is making that review seat fast enough to keep up with the machines feeding it, without turning it into a stamp.

What a fast review seat looks like

We have written before about the eight-second decision, and the number is not rhetorical. It is roughly the time a competent reviewer needs to accept or reject a change when everything they need to know is in front of them and nothing they do not need is. Almost no review tool is built for that number. They are built for the twenty-minute review, which was designed for human-written code where the intent had to be reverse-engineered from the diff.

Agent-written code has a property human-written code never had: the intent already exists in writing, because somebody typed it into a prompt. The specification, the plan, the reason for every choice, the tests it ran, the things it decided not to do. All of that is sitting in a transcript that the review tool throws away. The review seat is slow because it is being asked to rediscover information the system already had.

What the reviewer needs in front of them for an eight-second decision, and where each piece already exists today (nowhere in the pull request).

Five things. The request, the plan, the verification, the blast radius, and the list of things the agent decided it was not sure about. That last one is the one nobody surfaces and the one that makes the decision fast, because a reviewer who can see "the agent was unsure about the retry logic and left it as before" knows exactly where to spend their eight seconds.

The harness study that hit Hacker News this week is interesting in this light. Nine researchers ran 176 matched configurations across four models on SWE-Bench Verified and Terminal-Bench, varying planning, action space and context management. One of their findings is that for stronger models, explicit planning stopped improving accuracy and became mainly a cost saver. Read as a review problem rather than a benchmark problem, that says the plan is cheap to produce and does not hurt the agent. Which means there is no excuse for it not being attached to the pull request, because it is the single most useful artefact a reviewer could have and the model will write it for nothing.

What to do on Monday

Measure the queue. Not the cycle time, which averages away the problem, but the number of changes waiting for a human and how long the oldest one has waited. If that number is growing week over week, you have the disease and no amount of model upgrades will treat it.

Then re-engineer the seat rather than removing it. Attach the prompt and the plan to the pull request, automatically, as the description. Make the agent state what it verified and how, in a fixed format, at the bottom. Make it list what it was unsure about. Route by blast radius: a change that touches one file and no data can go to a fast lane with a fast reviewer; a change that touches billing goes to a slow lane with a senior one. Give reviewers a budget of decisions per day rather than a queue of infinite length, and watch what happens to the quality of the decisions.

One more thing worth stealing from manufacturing, since the factory metaphor is doing so much work here. A line with a bottleneck station is run at the pace of the bottleneck, on purpose, and the upstream stations are told to stop rather than pile up inventory. Software teams do the opposite: they celebrate the pile. A hundred open pull requests is not throughput. It is inventory, and inventory decays, because the codebase underneath it keeps moving and every day a change waits is a day closer to a merge conflict and a re-review. Cap the queue. Let the agents idle. It feels wrong for about a week.

And resist the auto-approve until you have done all of that. Not because the tools are bad but because auto-approving forty percent of changes into a system with no fast human lane for the other sixty is how you get a queue of the hardest sixty percent with the least context, reviewed by the most tired people. The machines will keep getting faster on a schedule you do not control. The seat is the only part of the pipeline you actually own.

The engineers were never the slow part. The place where a person has to say yes is the slow part, and it was always going to be, and the job now is to make yes cheap without making it meaningless.

Sources

• Towards Self-Driving Codebases (detail.dev) (HN, 119 points, 17 Sep 2026)
• Please stop using stupid models (t3.gg, video) (18 Sep 2026)
• Anthropic researchers are quitting... and now we know why (Fireship, video) (15 Sep 2026; the Macroscope figure is from the sponsor segment, so a vendor claim)
• An Empirical Study of Harness Design for Coding Agents (arXiv 2609.20804) (HN, 217 points, 18 Sep 2026)
• Levelbrook: the eight-second review (earlier playbook piece)

Originally published on the Levelbrook playbook. Levelbrook is a principal-led Rails and AI-systems consultancy; the playbook is where we write down what we see.

The scoreboard that changes every six weeks

Here is what the last ten days sounded like if you are an engineer with a job and a family and a finite amount of evening.

Theo Browne released a video saying that if you cannot feel the difference between this generation's frontier models and the last one, you suck at prompting, and he meant it kindly but he meant it. ThePrimeagen spent an episode on how a company that invented the transformer and the tensor processing unit is, on the coding leaderboards he showed, being beaten by a lab with a few hundred employees, and then left one of its models on a trivial bug for forty minutes and watched it spend 330 million tokens reading the same file. AI Explained walked through six axes on which the labs say capability will keep improving, none of them near saturation, and quoted a researcher saying there is a large gap between how fast progress looks from inside and from outside. A model called Astra that could do things nothing before it could. Another called Fable that was the safe choice three weeks ago. A Chinese open-weight model that is months old and beating both on some table.

If you tried to keep up with that, you did not sleep and you learned nothing durable, because almost every specific fact in the paragraph above will be wrong by November. That is not a prediction about any particular model. It is a description of the scoreboard. It reshuffles every six to eight weeks and it has done so for two years and every lab in the AI Explained video says it will keep doing so.

So the anxiety is real and the scoreboard is real and the two have almost nothing to do with each other. The question worth an evening is which things you can learn now that will still be true when the scoreboard has reshuffled four more times.

The half-life of what you know

Sort what an engineer learns about these tools by how long it stays true.

At the bottom, decaying in weeks: which model is best at which task, which one reads the same file twenty-six times, which one is over-eager and which one is lazy, the prompt phrasing that makes a particular model stop apologising. This is what most of the content is about, because it changes constantly and change is what content is made of. Learning it is not worthless. Learning it is maintenance, like knowing this quarter's prices.

In the middle, decaying in months: the harness. Which tool, which agent runtime, which instruction file format, which permission model. The study we wrote about earlier this week found that the harness is a first-class variable, more important than the leaderboards admit, and that is true. It is also true that the harnesses are being rewritten as fast as the models, and that the changelog entry that got seven hundred points this week was about one tool starting to read another tool's config file.

At the top, not decaying at all as far as anyone can tell: five things. We will name them and then argue for each one, because the argument is where the reassurance actually lives.

What an engineer learns about AI tools, sorted by how long it stays true. Almost all the content is about the bottom row. Durations illustrative.

The five things

Specification. The single skill that has appreciated most in two years is the ability to say, precisely and in writing, what a system should do, including what it should refuse to do and every exception. This used to be a skill people apologised for having, because the code was the specification and writing it down twice was waste. Now the specification is the input to the machine that writes the code, and the quality of the output is bounded by the quality of that input in a way that no model upgrade changes. Every team that got disappointing results from agents this year got them from underspecified tasks. The detail.dev post says the most valuable engineering work is going to be having good ideas, and an idea that cannot be specified is not yet an idea.

Verification. Knowing whether a thing works, and being able to prove it to someone else, is the skill the machines are worst at and the one they generate the most demand for. Every agent-written change needs someone who can say what test would catch the failure, whether the test that was written is that test, and whether the green result means what it appears to mean. Dan Luu's essay this week is on the front page for a reason: there is no point at which turning your brain off works, and verification is the name for the brain being on. This is also, not coincidentally, the skill Theo was actually describing when he said people prompt badly. The people who get good results from strong models are the people who can tell when the result is bad.

Judgement. Which of the forty things the model could do is the one worth doing. Which of the three approaches it offered is the one that will still be fine in a year. When to stop. When the obvious solution is obvious because it is wrong. Every essay on this site comes back to this because every failure we have watched comes back to it: the model did something competent that nobody should have asked for. Judgement is slow to build, does not transfer from a video, and is the entire reason a senior engineer is paid more than a junior one with the same tools.

Domain intimacy. Knowing the business, the customers, the forty exceptions and why the freight claim in 2023 changed how one account is handled. We have argued at length that companies do not have processes, they have habits, and that the habits live in heads. The heads are the moat. A model with the rumour builds the obvious version of your product; the person who knows why the obvious version fails for the second-largest customer is the person the swarm cannot replace, and that person is more valuable this year than last, not less.

The seat. The ability to occupy the approval seat well: to look at a model's output, know where to spend eight seconds, sign for it, and stand behind the signature. This is the one that combines the other four and it is the one the organisation will pay for when the volume of model output has outrun everyone's ability to read it. It is also the one nobody teaches, because until eighteen months ago the person in the seat had also done the work.

What this means for the evening

Spend it differently.

Stop trying to keep up with the scoreboard and start reading it the way you read exchange rates: glance, note the direction, move on. Pick one strong model and one harness and learn them properly for a quarter rather than four of each badly. Theo's floor-and-ceiling point is correct and it is also an argument for depth: you learn where a model's floor is by using it on dull tasks for weeks, not by watching someone else's demo of its ceiling.

Then spend the real time on the five. Write the specification for the next thing your team builds before anyone, human or model, writes a line, and notice how much you did not know. Write the test that would catch the failure before you look at the agent's tests. When an agent offers three approaches, write down why you picked one, and read your reasons back in a month. Learn the part of the business your team pretends is simple. Volunteer for the review seat that everyone else is avoiding because the queue is long, and get fast at it, because that queue is the most important place in the company and almost nobody wants to sit there.

The trade an engineer can make this quarter. The left column is what the content wants you to do; the right is what compounds.

The honest caveat

It is possible to hide from the tools behind this list, and some people will. Specification and verification and judgement are worth nothing if you refuse to use the machines that make them valuable, and the engineer who has not sat in front of a frontier model for a hundred hours does not actually know where its floor is and cannot occupy the seat. The scoreboard is not the point, but the tools are, and the argument here is for depth with them rather than distance from them.

The researchers in the AI Explained video may be right that the gap between inside and outside is large and that things will move faster than the outside expects. If so, the scoreboard will reshuffle faster, not slower, and the half-life of everything in the bottom two rows gets shorter. The five things at the top do not get shorter. They get more expensive, because the volume of machine output that needs specifying, verifying, judging, situating and signing for goes up with every release, and the number of people who can do those things well does not.

You are not behind. You have been watching the part of the screen that changes. Look at the part that does not, and get good at it while everyone else is refreshing the leaderboard.

Sources

• Please stop using stupid models (t3.gg, video) (18 Sep 2026)
• How are they losing so bad (ThePrimeagen, video) (15 Sep 2026; the token count and the model timeline are as reported on stream)
• What AI Researchers Saw, Before Their Demand to 'Pace' AI (AI Explained, video) (16 Sep 2026)
• Towards Self-Driving Codebases (detail.dev) (HN, 119 points, 17 Sep 2026)
• There's no point at which turning your brain off will work (Dan Luu) (HN, 196 points, 18 Sep 2026)

Originally published on the Levelbrook playbook. Levelbrook is a principal-led Rails and AI-systems consultancy; the playbook is where we write down what we see.