Power-law scaling, a central concept in critical phenomena, is found to be
useful in deep learning, where optimized test errors on handwritten digit
examples converge as a power-law to zero with database size. For rapid decision
making with one training epoch, each example is presented only once to the
trained network, the power-law exponent increased with the number of hidden
layers. For the largest dataset, the obtained test error was estimated to be in
the proximity of state-of-the-art algorithms for large epoch numbers. Power-law
scaling assists with key challenges found in current artificial intelligence
applications and facilitates an a priori dataset size estimation to achieve a
desired test accuracy. It establishes a benchmark for measuring training
complexity and a quantitative hierarchy of machine learning tasks and
algorithms.

使用幂律缩放在深度学习中有助于处理当前人工智能应用中的关键挑战，可用于测量训练复杂性和机器学习任务算法的量化层次，并建立数据集大小估计的基准以实现期望的测试准确性。

利用幂律缩放应对人工智能面临的关键挑战

Power-law Scaling to Assist with Key Challenges in Artificial Intelligence

The test loss of well-trained neural networks often follows precise power-law
scaling relations with either the size of the training dataset or the number of
parameters in the network. We propose a theory that explains and connects these
scaling laws. We identify variance-limited and resolution-limited scaling
behavior for both dataset and model size, for a total of four scaling regimes.
The variance-limited scaling follows simply from the existence of a
well-behaved infinite data or infinite width limit, while the
resolution-limited regime can be explained by positing that models are
effectively resolving a smooth data manifold. In the large width limit, this
can be equivalently obtained from the spectrum of certain kernels, and we
present evidence that large width and large dataset resolution-limited scaling
exponents are related by a duality. We exhibit all four scaling regimes in the
controlled setting of large random feature and pretrained models and test the
predictions empirically on a range of standard architectures and datasets. We
also observe several empirical relationships between datasets and scaling
exponents: super-classing image tasks does not change exponents, while changing
input distribution (via changing datasets or adding noise) has a strong effect.
We further explore the effect of architecture aspect ratio on scaling
exponents.

该研究提出了一种理论，解释并连接训练数据集大小和网络参数数量与已训练神经网络的测试损失之间的精确定义的幂律关系，并通过说明数据流形和一些核的频谱之间的等效性来解释了分辨率有限的缩放行为。

神经缩放定律解释

Explaining Neural Scaling Laws

Deep learning (DL) creates impactful advances following a virtuous recipe:
model architecture search, creating large training data sets, and scaling
computation. It is widely believed that growing training sets and models should
improve accuracy and result in better products. As DL application domains grow,
we would like a deeper understanding of the relationships between training set
size, computational scale, and model accuracy improvements to advance the
state-of-the-art.
This paper presents a large scale empirical characterization of
generalization error and model size growth as training sets grow. We introduce
a methodology for this measurement and test four machine learning domains:
machine translation, language modeling, image processing, and speech
recognition. Our empirical results show power-law generalization error scaling
across a breadth of factors, resulting in power-law exponents---the "steepness"
of the learning curve---yet to be explained by theoretical work. Further, model
improvements only shift the error but do not appear to affect the power-law
exponent. We also show that model size scales sublinearly with data size. These
scaling relationships have significant implications on deep learning research,
practice, and systems. They can assist model debugging, setting accuracy
targets, and decisions about data set growth. They can also guide computing
system design and underscore the importance of continued computational scaling.

本文采用实证方法针对机器翻译、语言建模、图像处理和语音识别等 4 个机器学习领域的数据，研究训练集大小、模型规模与推广误差之间的关系，结果表明推广误差遵循幂定律缩放，且模型改进只改变误差而不影响幂指数。此外，模型大小随数据规模的增大缩小，这些研究对于深度学习的研究、实践和系统设计具有重要意义。

深度学习的扩展是可预测的，实证的

Deep Learning Scaling is Predictable, Empirically

Patterns of deliberate human activity and behavior are of utmost importance
in areas as diverse as disease spread, resource allocation, and emergency
response. Because of its widespread availability and use, e-mail correspondence
provides an attractive proxy for studying human activity. Recently, it was
reported that the probability density for the inter-event time $\tau$ between
consecutively sent e-mails decays asymptotically as $\tau^{-\alpha}$, with
$\alpha \approx 1$. The slower than exponential decay of the inter-event time
distribution suggests that deliberate human activity is inherently
non-Poissonian. Here, we demonstrate that the approximate power-law scaling of
the inter-event time distribution is a consequence of circadian and weekly
cycles of human activity. We propose a cascading non-homogeneous Poisson
process which explicitly integrates these periodic patterns in activity with an
individual's tendency to continue participating in an activity. Using standard
statistical techniques, we show that our model is consistent with the empirical
data. Our findings may also provide insight into the origins of heavy-tailed
distributions in other complex systems.

通过电子邮件通信的间隔时间，作者估算人类活动的非泊松性与昼夜节律和周律有关，并提出了一个级联的非齐次泊松过程来模拟这样的间隔时间，并证明了该模型能够很好地解释实际数据。