Anthropic Study Finds AI Coding Tools Cut Developer Skill Retention by 17%

Lawrence Jengar
Jan 29, 2026 19:57

New randomized trial from Anthropic reveals developers using AI assistance scored nearly two letter grades lower on coding comprehension tests, raising workforce development concerns.

Developers who rely on AI assistants to write code score 17% lower on comprehension tests than those who code manually, according to a randomized controlled trial published by Anthropic on January 29, 2026. The gap—equivalent to nearly two letter grades—raises pointed questions about workforce development as 82% of developers now use AI tools daily.

The study tracked 52 junior software engineers learning a new Python library called Trio. Participants with AI access averaged 50% on a follow-up quiz, compared to 67% for the hand-coding group. Debugging skills showed the steepest decline, a particularly concerning finding given that catching AI-generated errors remains a critical human oversight function.

Speed Gains Weren’t Statistically Significant

Here’s what might surprise productivity hawks: the AI group finished only about two minutes faster on average, and that difference didn’t reach statistical significance. Several participants spent up to 11 minutes—30% of their allotted time—just composing queries to the AI assistant.

This complicates the prevailing narrative around AI coding tools. Anthropic’s own earlier research found AI can reduce task completion time by 80% for work where developers already have relevant skills. But when learning something new? The productivity picture gets murkier.

How You Use AI Matters More Than Whether You Use It

The researchers identified distinct interaction patterns that predicted outcomes. Developers who scored below 40% typically fell into three traps: fully delegating code to AI, starting independently but progressively offloading work, or using AI as a debugging crutch without building understanding.

Higher performers—averaging 65% or above—took different approaches. Some generated code first, then asked follow-up questions to understand what they’d produced. Others requested explanations alongside generated code. The fastest high-scoring group asked only conceptual questions, then coded independently while troubleshooting their own errors.

The pattern suggests cognitive struggle has value. Participants who encountered more errors and resolved them independently showed stronger debugging skills afterward.

Workforce Implications

The findings land amid explosive growth in AI-assisted development. The global AI in education market is projected to hit $32.27 billion by 2030, growing at 31.2% annually. Major platforms including Claude Code and ChatGPT have already introduced “learning modes” designed to preserve skill development—an acknowledgment that the problem Anthropic documented isn’t theoretical.

For engineering managers, the study suggests aggressive AI deployment may create a capability gap. Junior developers optimizing for speed could miss the foundational debugging skills needed to validate AI-generated code in production environments. The researchers note this setup differs from agentic coding products like Claude Code, where impacts on skill development “are likely to be more pronounced.”

The study has limitations—small sample size, immediate rather than long-term assessment, and focus on a single programming domain. But it offers early evidence that productivity gains and skill development may pull in opposite directions, at least for workers learning new capabilities. Companies betting heavily on AI-augmented development might want to factor that trade-off into their training strategies.

Image source: Shutterstock