← Back to homepage

Stop Testing What the Agent Does Best

An Anthropic hiring manager has dropped the coding interview from his own hiring and says he has not yet found what replaces it. The timed coding test measures convergent thinking, and agents now do that well. I would put four things in its place: a design conversation, a session working with an agent, a structured look at how someone works with others, and a short check of everyday developer skills. None of them has been validated yet.

At the end of October, Felix Rieseberg, an engineering manager at Anthropic, opened a blog post with this: “This month, I laid the traditional coding interview to rest. As a hiring manager at Anthropic, I stopped asking candidates to do them.”

He was fair to the old test. Defining a problem quickly and finding an elegant solution “was a weak but useful predictor for how successful you’d be on my teams”, he wrote, and small challenges in real code “did tell us something meaningful”. Then: “That’s no longer true. There’s no more point in giving you a little React challenge than there is in quizzing people about assembly; Claude is now better at both.”

The newsletter headline that followed said Anthropic had killed the coding interview. Rieseberg’s post says something narrower. It is about his own hiring, and its last paragraph admits he has not worked out the replacement. For now he is scanning résumés and portfolios “for little prototypes that exhibit taste, empathy, and principled thinking”, and he asks anyone with a better idea to tell him.

I have a better idea, or at least a more complete one. First, though, what exactly stopped working?

What the old test measured

J. P. Guilford, the psychologist who coined the terms convergent and divergent thinking, put the difference plainly in 1959: “Convergent thinking proceeds toward a restricted answer or solution.” In his 1967 book the test-taker “must converge upon the one right answer”. Divergent production is the other kind, where the emphasis is on “variety and quantity of output from the same source”.

The timed coding exercise is a convergent test. The problem is fully specified, there is a right answer or a small family of them, and the candidate is scored on getting there within the hour. That is exactly the kind of work a coding agent is now good at.

Anthropic has the clearest evidence, because it watched its own test fail. Tristan Hume describes a performance-engineering take-home that more than 1,000 candidates had completed since early 2024. Given the same time limit, Claude Opus 4 “outperformed most human applicants”, and Opus 4.5 then matched even the strongest. “Humans can still outperform models when given unlimited time,” Hume writes, “but under the constraints of the take-home test, we no longer had a way to distinguish between the output of our top candidates and our most capable model.”

Canva reached the same point by a cheaper route. When it ran its Computer Science Fundamentals questions through AI tools, they “produced correct, well-documented solutions in seconds, often without requiring any follow-up prompts or clarification”.

One caution from Guilford himself. He noted that people often do “much divergent production on the way to a convergent answer”. A good engineer solving an interview puzzle is still exploring options. The trouble is that the exercise only scores where they end up, and the agent ends up there faster.

What the job looks like now

Hume is candid about what his team’s work has become: “tough debugging, systems design, performance analysis, figuring out how to verify the correctness of our systems, and figuring out how to make Claude’s code simpler and more elegant.” He adds the inconvenient part: “Unfortunately these things are tough to test in an objective way without a lot of time or common context.”

The developer surveys say the same thing from the other side. In Stack Overflow’s 2025 survey, the most common frustration with AI tools, cited by 66% of developers who answered the question, was “AI solutions that are almost right, but not quite”. The 2025 DORA report tells leaders to teach teams “to critically guide, evaluate, and validate AI-generated work, rather than simply encouraging usage”. The scarce skill is judging output, and the old interview gave no signal on it.

What other companies are doing

Companies have started to move, in different directions. Canva piloted an “AI-Assisted Coding” interview in place of its fundamentals screen for backend and frontend roles, set open problems (“Build a control system for managing aircraft takeoffs and landings at a busy airport”) and insists candidates use AI. Meta’s careers page now says many of its interviews include an assistant and “Candidates are expected to use this AI assistant as part of the interview.” Google has moved partly the other way. Sundar Pichai said last year the company would “introduce at least one round of in-person interviews for people just to make sure the fundamentals are there”.

Anthropic’s written guidance for candidates, last updated in July 2025, says take-homes and live interviews are done without AI unless Anthropic says otherwise. Hume’s take-home uses that exception, and Rieseberg has dropped the coding round altogether. The guidance mentions neither.

Many companies have not changed. In Karat’s 2025-2026 survey of 400 engineering leaders, 62% said they do not allow candidates to use AI in interviews, and 71% said AI is making technical skills harder to assess. Karat sells AI-enabled interviews, so read those numbers with that in mind. Even so, both numbers point the same way: the test is getting harder to read, and most firms have not changed the rules.

What the evidence says about interviews

The figures most people quote come from Schmidt and Hunter’s 1998 review, which put work samples at .54 and structured interviews at .51 as predictors of job performance. In 2022 Paul Sackett and colleagues re-analysed the same literature and argued that many earlier estimates had been inflated by over-correcting for range restriction. Their headline is that “structured interviews emerged as the top-ranked selection procedure”, at .42. Work samples dropped to .33. That figure comes from a 2005 meta-analysis by Roth and colleagues; the old .54 came from a study that pre-dated meta-analysis itself. Unstructured interviews came in at .19.

Two things follow. First, hands-on work samples were never as strong a predictor as their reputation, and none of the studies looked at coding tests specifically. Second, the best-supported thing you can do is ask every candidate the same job-relevant questions and score the answers against the same rubric. Sackett’s team also warn that structured interviews vary a lot from one study to the next, so a badly built one carries “a higher risk of obtaining a lower value”. None of these studies looked at software engineers working with AI. They tell you how to build an assessment, not which tasks to put in it.

That matters, because the new formats come with very little evidence. Canva says its AI-assisted interviews are “providing strong predictive signals about candidate performance” but publishes no data. Hume says early scores on his newest take-home correlate well with the calibre of candidates’ past work, but gives no coefficients. I have not found a published validity figure for any AI-era interview format.

What I would put in its place

The aim is to stop scoring convergent output and start scoring the judgement around it. There are four parts, and they can sit side by side in one loop.

A design conversation

Give the candidate a problem with no single answer and ask them to design a solution, talk it through and draw it. Then push on the trade-offs. Why this store and not that one, and what breaks first when traffic triples? Many companies already run a system design round, and this is close to it. The difference is that it carries more of the weight, and the candidate is scored on how they reason about options, not on reaching a known answer.

There is a catch here. In a randomised study of 48 computer science students, Behroozi and colleagues found that 61.5% failed a whiteboard problem when an interviewer watched while they thought aloud, against 36.3% of those who solved it alone. The interviewer was silent and the stakes were low, and the authors call their recommendations hypotheses. Still, a design conversation also has an audience. Their suggestion fits what I want: treat the problem as “a shared vehicle through which the interviewer and candidate engage in a conversational dialogue”. The interviewer should take part in the conversation, not just watch it.

A session with a coding agent

Coding still matters, but what I want to see has changed. What context does the candidate give the agent before asking it for anything? Do they break the problem into pieces it can handle? When the code comes back, how do they decide whether it is right? Canva found its strongest candidates “asked thoughtful clarifying questions”, used AI “for well-defined subtasks while maintaining control of the overall solution” and critically reviewed what came back. Candidates with little AI experience often struggled, “not because they couldn’t code, but because they lacked the judgment to guide AI effectively”. One rule from Canva’s interviewer guide is worth copying: when the interviewer asks a question, the candidate gives their own view before asking the AI.

Hume names the risk in this format. A candidate steering an agent may end up “understanding what Claude did only after the fact. The dominant strategy might become sitting back and watching.” So the task has to need steering. Hand them something where the agent’s first answer is plausible and wrong, and watch whether they notice. I wrote about why that checking is the hard part in What It Takes to Trust Agent Output.

How they work with others

Team fit still matters, and I think it should. But I am not looking for heroes. I want people who explain their thinking and ask for help early, for the reasons I set out in The Myth of the Heroic Developer.

“Fit” is also where interviews go most wrong. Lauren Rivera spent nine months observing a hiring committee at one elite firm and interviewed 120 people who hired for banks, law firms and consultancies. More than half rated fit as the most important criterion at interview, above analytical thinking and communication, and fit mostly meant shared leisure pursuits and backgrounds. One of her interviewees put it bluntly: “You are basically hiring yourself. This is not an objective process.” In one meeting she observed, a candidate who did well on the case was turned down as “too intellectual” because he was “very into 18th-century literature and avant-garde film”.

Rivera separates similarity in work styles, which she says can be beneficial, from similarity in play styles. That is the line I would hold. Write down the behaviours that matter on your team and ask every candidate about them in the same way. How did they handle a reviewer who disagreed with them? Whose work did they mention without being asked? Score the answers. Do not score whether you would enjoy a beer with them.

The everyday basics

It still surprises me how often developers struggle with work we would once have called simply part of the job: a git merge, finding their way round the IDE. I have seen someone hand Claude a merge that was two files and a few lines.

Models are not reliable at this either. On Merge-Bench, which gave mid-2025 models 7,938 real merge conflicts, each capped at 20 lines a side and with no agent tooling, “the best models correctly resolve less than 60% of merge conflicts”, and the authors say the size cap makes that “an over-estimate of performance on all merge problems”. So a short hands-on task belongs in the loop, such as resolving a small conflict or tracking down a failing test in an unfamiliar codebase. I am not timing how fast they type commands. I want to see that they can read what the tools are telling them and confirm the result, because sooner or later the agent will hand them something wrong.

What I cannot claim

None of this is proven. The four parts are my own answer to Rieseberg’s open question. The research I have relied on supports structure and job relevance in general. It does not show that a design conversation, an agent session or a merge exercise predicts how well a software engineer will do. I have found nothing published that shows it, and I have not measured it either.

So if I run this, I want to measure it. That means keeping a separate score for each part and checking those scores a year later against how people actually performed. None of the redesigns I looked at has published that check.

There is a link here to The First Rung, where I worried about who will be senior in five years. Rieseberg’s post is advice for engineers without status, which usually means people early in their careers, and its title, “Hunger beats tenure”, is a hopeful one. An interview that scores judgement instead of puzzle-solving gives a newer engineer with good instincts a fairer chance against someone who has simply done more puzzles. It only does that if we keep the structure and leave the beer test out.