In July 2025, METR (Model Evaluation and Threat Research) published one of the more uncomfortable studies in the AI-productivity conversation. They ran a randomised controlled trial with experienced open-source developers, working on their own real repositories, using current AI coding tools. Before the study, the developers predicted the tools would make them roughly 24% faster. Afterwards, most still believed they'd been about 20% faster.
They hadn't. Measured against a control group doing the same work without AI assistance, the AI-assisted developers were 19% slower.
"Believed 20% faster. Measured 19% slower." That 39-point gap between perception and reality is the whole finding.
The study's own analysis pointed to a few consistent causes: developers spent real time prompting, reviewing, and correcting AI output, and that time is easy to undercount because it doesn't feel like the friction of "real coding". Verifying a plausible-looking AI suggestion against an unfamiliar, complex codebase took longer than developers expected, and the tools were often being used on the kind of high-context, high-stakes work where getting it wrong is expensive to unwind.
None of this means the tools don't work. It means confident use and effective use are two different things, and the study measured the gap between them directly, rather than relying on developers' own sense of how well things were going.
A follow-up study in May 2026 revisited the question with technical workers whose AI-assisted workflows had had time to mature, rather than developers using the tools for the first time under study conditions. The result flipped: workers reported 1.4 to 2 times the output on comparable tasks.
The tool hadn't fundamentally changed in ten months. What had changed was the habit: knowing when to trust a suggestion and when to slow down and verify it, how to structure a task so the AI has enough context to be useful, and how to review output like you'd review a colleague's work rather than rubber-stamping it.
Reading these two studies together, the actionable finding isn't "AI helps" or "AI doesn't help" — it's that the technique separating the two outcomes is learnable, and it's exactly the kind of thing that's hard to pick up from documentation alone, because you don't usually find out you're doing it wrong until someone watching you work points it out.
That's the premise behind live 1-to-1 coaching rather than a recorded course: a specialist watching you work in your own repo can see the gap between confident and effective use directly, in real time, and close it far faster than trial and error will.
Sources: METR, "Measuring the Impact of Early 2025 AI on Experienced Open-Source Developer Productivity" (2025) and METR, 2026 AI usage follow-up survey.