BUILT BY YAHOO

When AI Can Write the Code, How Do You Interview the Engineer?

Imagine walking into a software engineering interview and being asked to build an app from scratch. The catch: You don’t have nearly enough time to do it yourself. That’s the point.

Traditionally, technical interviews have tested whether engineers can solve coding problems themselves, whether on a whiteboard, in a shared editor, or through algorithmic challenges. Coding agents are complicating that model.

“The coding interview is broken for everybody,” says Kat Styons, Senior Director of Software Development Engineering at Yahoo. “Agents are great at solving these.”

The coding interview is broken for everybody

If an agent can solve the coding problem, Styons’ team needs the interview to answer a different question: How do you evaluate the engineer directing it? Their answer is an agentic coding interview designed to reveal what happens after the agent produces its first answer, especially when that answer starts to go wrong.

Make the agent essential

Styons’ team gives candidates a deliberately impossible assignment: build an enterprise messaging app in less time than a human could reasonably build it alone.

At first, Styons worried the experiment might be too easy. When she copied the entire assignment into a coding agent herself, it produced a credible first attempt. But that’s where the interview begins.

Interviewers let candidates take an initial pass, even if that means feeding the whole prompt to the agent. Then they keep changing the assignment, asking candidates to refine the app, reverse previous decisions, and add increasingly complex features.

After a few iterations, the agent begins to drift from what the candidate is trying to accomplish. That’s the moment Styons and her team are waiting for.

Wait for the agent to get lost

Once the agent goes off track, Styons says three patterns emerge.

Some candidates struggle to recover. They panic, lack a strategy for getting the agent back on course, or try to persuade the interviewer that its existing solution is good enough.

The second group is more revealing: they’re strong engineers who recognize the problem and know how to fix it. But when the agent frustrates them, they retreat to a familiar way of working.

“They flip back over to that coding pane, and start trying to investigate and fix it themselves,” Styons says. “And at that point, they lose the momentum in the session.”

The candidates who perform best have strategies for recovering without losing that momentum. They know how to manage the agent’s context and use the tool itself to investigate what went wrong.

Styons describes agentic development as a separation between authorship and ownership. Engineers may give up authorship of every line of code, but they remain accountable for its quality. That makes their judgment more important, not less.

Hire for judgment

Two qualities that always mattered to Styons have moved to the top of the list: curiosity and intellectual courage. Curious engineers question why an agent made a particular choice, seek a deeper understanding of what it produced, and proactively explore new tools and approaches. Intellectual courage shows up when something unexpected happens: the ability to persevere, form an opinion about the right path forward, and act on that judgment.

“Candidates with intellectual courage know how to commit to a decision and move forward, even when there are risks,” Styons says.

Those qualities matter because Styons believes the constraints on software development are shifting. As software becomes faster to produce, teams become less constrained by how much they can build and more by the quality of the decisions they make and how quickly they can learn from the results.

“Building code is no longer the bottleneck for us,” Styons says. She expects the new limiting factors to include “the taste and the judgment of the people on your team” and “how fast your organization can learn from failures and successes.”

‍

Key takeaways

AI is changing what technical interviews need to measure. When coding agents can handle much of the implementation, interviews can focus more on how engineers direct the work, diagnose problems, and recover when an agent goes off track.

Strong engineers need new ways of working. Technical skill remains essential, but teams need engineers who can recover from agent failures without reverting to manual coding.

Curiosity and intellectual courage are becoming hiring priorities. Engineers need the judgment to question, guide, and remain accountable for work increasingly authored by agents.

‍