Chinchilla rewrites the scaling laws

DeepMind shows most large models were under-trained; a 70B model on more data beats much larger ones.

What happened and why it matters

The compute-optimal scaling result meant the same compute budget could buy substantially more capability, an algorithmic efficiency gain that every subsequent frontier lab adopted.

Sources

• Training Compute-Optimal Large Language Models ↗ arXiv · 29 Mar 2022