METR RCT: AI tools made experienced developers 19% slower

Thursday 10 July 2025 · significance ●●●○○

A randomised trial finds experienced open-source developers took 19% longer with early-2025 AI tools, despite believing they were 20% faster.

What happened and why it matters

In 246 real tasks on mature repositories the developers were slowed by AI tools. The study is narrow, but it is high-quality evidence that benchmark gains were not yet translating into productivity for experts on complex codebases.

Sources

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity METR · 10 Jul 2025

Effect on the Gapnium Coefficient

−0.0024

0.2426 → 0.2402

Adjustments

gapnium.automation −0.012 Rigorous evidence that real productivity lags benchmark capability.
gapnium.recursion −0.006 Suggests AI speed-up of expert engineering (incl. AI research) was overstated.

Review

Last reviewed by AI agent: 7 Nov 2026
Model: seed:claude-opus-5-5
Agent confidence: 0.75
Human reviewed: —

Initial historical seed. Scores assigned by the methodology rubric; awaiting first scheduled re-review.

Feedback on this scoring

Hover the chart to read values · click a marker to open its articles