← Back to homepage

Random thoughts

Short, unfiled notes

Things worth writing down that don't need a full essay.

  • The no-AI control group is getting harder to keep

    The best-known trial of AI coding tools with experienced engineers is METR’s from 2025. Sixteen open-source developers worked through 246 tasks, each one randomly assigned to allow AI or forbid it. Tasks took 19% longer when AI was allowed, and the developers came away believing it had cut their time by about 20%. The design was clean. The same people did both kinds of work, and the answer came from time on the task, not from opinion.

    In February METR said it was changing that design. The follow-up started in August 2025 with 57 developers and more than 800 tasks, and the main reason it gave was not one I expected: more developers were choosing not to take part because they did not want to work without AI. METR gives no count. Among those who did take part, 30% to 50% said when surveyed that they were holding back some tasks for the same reason. One put it this way: “I avoid issues like AI can finish things in just 2 hours, but I have to spend 20 hours.”

    It’s easy to turn this into “developers won’t work without AI”, and some of the commentary did. METR’s claim is smaller. People withheld some tasks, and it says the selection seems to affect “a minority share of developers and of tasks”. It had also cut the pay from $150 an hour to $50, which it says “also likely contributed”. The new numbers lean towards a speed-up (tasks took an estimated 18% less time for the ten returning developers, and 4% less for the new ones), but both ranges include zero. METR calls the data “only very weak evidence” for how much things have improved since early 2025.

    What stays with me is which tasks went missing. They were the ones where developers expected AI to help most, so the study lost the cases with the largest expected effect before any task was assigned.

    Control groups have slipped before, in other ways. A Copilot field experiment at Microsoft ended in spring 2023, in its authors’ words, “as growing awareness of AI-assisted coding tools led the control group participants to seek access to Copilot”. Anthropic’s trial, the one I wrote about in The First Rung, had 35% non-compliance with its no-AI rule in the first two pilots. It held in the main study only after a change of platform and a check of screen recordings.

    I can’t tell from any of this whether it’s dependence or plain good sense. Nobody wants to spend 20 hours on what they believe is a 2-hour job, and I found no survey that asks developers whether they’d work without the tools.

    For anyone running a team the practical point is the same either way. Of the trials I looked at, the ones that held on to a control group were short sessions on fixed tasks, a few hours at most, and even one of those had to police it. That is a long way from real work. For work that runs over months the options are weaker: a staggered rollout, where the comparison lasts only weeks, self-report, which the 2025 trial showed can point the wrong way, or a comparison between people who took up the tool and people who didn’t. If you want a baseline for your own team, it is likely to be easier to get now than in a year.

  • Split the spec by audience, not by completeness

    A product spec usually tries to do two jobs at once: convince a human the feature is worth building, and tell a machine exactly how to build it. Those are different documents wearing the same file extension. The functional half (“what this does and why a user wants it”) is written in prose for someone deciding whether to approve the work. The technical half (“these are the interfaces, this is the data shape, here’s the file it lives in”) is written for whoever implements it, and increasingly that’s an agent, not a person.

    Stuffing both into one spec.md and handing the whole thing to a coding agent is a context-window problem dressed up as a documentation one. The agent doesn’t need the stakeholder framing, the user-empathy paragraph, or the business case; none of it changes what code it writes. It needs the contract. Every sentence of mission language it has to read past is a sentence diluting its attention from the part that actually constrains the output, and it’s easy for an agent to implement the wrong interface because the right detail was buried under paragraphs of customer-pain framing it didn’t need to read.

    The fix is two documents, not one: each agent in the pipeline reads only the one addressed to it. A planning agent breaking a feature into tasks wants the functional spec, intent and acceptance criteria and the edge cases a user would notice; an implementation agent wants the technical spec instead, types and endpoints and the directory a new file belongs in, nothing about why a user wants it. A review agent’s needs are narrower again: just the diff and the acceptance criteria, since it’s checking behaviour against intent rather than re-deriving the architecture. Same feature, several reading lists, none of them padded.

    This only works if someone keeps the two specs honest against each other, because splitting them creates exactly the drift problem that stale AGENTS.md files already have: a technical spec that quietly stops matching the functional one it was supposed to implement, with nothing forcing a cross-check. The cheap guard is structural rather than procedural. Put a single line at the top of the technical spec linking back to the functional requirement it satisfies, by ID, and make the review agent’s task include confirming that link still holds. That’s a much smaller ask than reading the whole functional document every time, and it catches the one failure mode that actually matters: implementation that’s internally consistent but has drifted from what was asked for.

    None of this is really about agents. It’s the same reason a well-run engineering org keeps the PRD and the design doc as separate artefacts instead of one sprawling file: different readers need different detail, and conflating them serves neither. Agents just make the cost of ignoring that lesson visible immediately, because an overloaded context window fails today, in front of you, instead of quietly next quarter when the PM and the engineer turn out to have read the same paragraph and understood two different features.

  • Your AGENTS.md is policy, not documentation

    Two different studies on AGENTS.md files landed this year, and it’s worth keeping their findings separate. One is a length problem: too much context dilutes an agent’s attention, the same way a bloated onboarding doc loses a new hire. The other is a staleness problem, and it’s a different animal entirely. Wrong context isn’t just useless, it’s actively worse than no context at all, because the agent still follows it.

    That distinction matters because of what AGENTS.md actually is. Ordinary documentation describes: “the app uses Postgres” is a claim a person can check against the code and shrug off if it’s slightly wrong. AGENTS.md is behavioural: “never modify generated files, always run the migration script before touching schemas” is an instruction an agent executes. That makes it closer to configuration, or even source code, than to a README, and it deserves the same question we ask of source code: not just whether it reads well, but whether it’s correct.

    The asymmetry is what makes stale AGENTS.md files dangerous in a way stale docs aren’t. A stale function usually fails loudly: it doesn’t compile, or a test catches it. A stale instruction doesn’t fail at all. The agent reads it, obeys it, produces plausible code that satisfies the obsolete rule, passes the tests that don’t know any better, and merges. Nothing trips.

    The evidence backs this up from two directions. A study of 466 open-source AGENTS.md files found five completely different writing styles in use, no shared structure, and roughly half the files unchanged since the day they were created, quietly drifting out of sync with a codebase that kept moving. A follow-up study testing agents against real tasks found that AI-generated context files actually made agents worse, cutting success by about 3%, while short, human-written ones improved it by about 4%, and both approaches cost more in inference either way. Bigger and better-written aren’t the same fix. One is a size problem. The other is a trust problem.

    So the answer isn’t a longer, more careful AGENTS.md. It’s a narrower one. Write down only what the code genuinely can’t tell an agent on its own, the conventions no config file expresses, the invariant nobody wants violated. Then treat it like any other claim about the system: revisit it when the thing it describes changes, not on a schedule and not never. Everything else belongs in package.json, not in prose.

  • AI is eating its own documentation

    We have decent tools for catching code decay. Linters, static analysis, test coverage and complexity metrics can all flag AI-generated code as it starts to rot. Documentation has nothing like that, and I think it is decaying faster.

    The pattern is easy to spot once you look for it. Someone asks an assistant to document a module. What comes back is long, confident and mostly plausible. Nobody reads it closely because it looks finished. A few months later the code has moved on and the docs haven’t. The next assistant is handed those stale pages as context and writes more documentation on top of them, a little more wrong each time.

    It’s an ouroboros, with AI eating its own output. Every pass adds bulk and loses precision, and because the prose reads so fluently, the drift is hard to see. People stop trusting the docs and go back to reading the code. The documentation ends up existing mainly to be fed to the next model.

    This deserves serious effort, and I’d put it ahead of a lot of code-quality work. Broken code usually trips a test or a build. A wrong paragraph just sits there, misleading people and machines alike until someone gets burned.

    In practice, documentation needs real time in the plan, not whatever is left at the end of a story. Short beats comprehensive: one page a new engineer can read in five minutes and trust is worth more than twenty generated ones. Docs get reviewed like code, have a named owner, and get deleted when they stop being true. The parts that explain why the system is the way it is should be written, or at least rewritten, by someone who understands it, because that is exactly what a model can’t recover from the code.

    Good documentation used to be a courtesy to the next person. Now it is also the input to every AI tool the team uses, so its quality compounds. That works in both directions.

  • Prompt precision as a design skill

    A prompt that works on the first try isn’t luck. It usually means you already had a precise mental model of the thing you were building before you typed a word, and you folded that model into one instruction that holds together.

    Software craftsmanship has always rewarded people who can state a problem clearly before touching a keyboard. Prompting just makes that skill visible in real time. When an AI returns a subtly wrong answer, the fault is often in the prompt rather than the model. The prompt is reflecting a gap in your own specification, one you would have hit anyway a few commits later.

    A prompt that lands cleanly on the first pass also has to carry the constraints a careless implementation would skip, like type safety and proper error handling. Writing those down before generation is system design, not a shortcut around it. Vague prompts produce vague code. So you cut the fluff and say exactly what you mean, which is the same habit that makes code readable in the first place.

    None of this replaces the underlying skill. Knowing what to ask for, and knowing whether what came back is actually right, still depends on how well you understand the problem. What a well-scoped prompt buys you is an AI acting as an extension of that understanding rather than a chat partner you keep having to correct.