RE-Bench: agents vs. human ML researchers

Friday 22 November 2024 · significance ●●○○○

METR finds AI agents beat human experts on ML research-engineering tasks at 2 hours, but humans win decisively given 8+ hours.

What happened and why it matters

RE-Bench directly measures the capability most relevant to recursive self-improvement. Agents were faster and better on short budgets; humans scaled much better with time — a measured gap rather than a solved problem.

Sources

RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts arXiv · 22 Nov 2024

Effect on the Gapnium Coefficient

+0.0012

0.1961 → 0.1973

Adjustments

gapnium.recursion +0.010 Agents already competitive with experts on short ML-engineering tasks.
gapnium.autonomy −0.004 Agents scale poorly with longer time budgets.

Review

Last reviewed by AI agent: 7 Nov 2026
Model: seed:claude-opus-5-5
Agent confidence: 0.75
Human reviewed: —

Initial historical seed. Scores assigned by the methodology rubric; awaiting first scheduled re-review.

Feedback on this scoring

Hover the chart to read values · click a marker to open its articles