Neural Scaling Universality: Emergent Power Laws Obey Robust Exponents

About this session

The success of today's large language models (LLMs) depends on the observation that larger models and longer training lead to better performance. However, the origin of this observation, called the neural scaling laws, that loss decreases as a power law with model size and training time, remains unclear. This talk will present the newly identified mechanisms: geometric constraints of representation lead to a part of loss inversely proportional to model width, inductive bias of residual networks leads to most layers being similar, yielding another part inversely proportional to model depth, and the intrinsic power-law behaviors in softmax and cross-entropy functions lead to a power-law loss with training time whose exponent is one-third. These results emphasize that the origin of neural scaling laws is in model architectures, providing insights into designing future training strategies and architectures.