Hoffmann et al. (2022) propose three methods for estimating a compute-optimal
scaling law. We attempt to replicate their third estimation procedure, which
involves fitting a parametric loss function to a reconstruction of data from
their plots. We find that the reported estimates are inconsistent with their
first two estimation methods, fail at fitting the extracted data, and report
implausibly narrow confidence intervals--intervals this narrow would require
over 600,000 experiments, while they likely only ran fewer than 500. In
contrast, our rederivation of the scaling law using the third approach yields
results that are compatible with the findings from the first two estimation
procedures described by Hoffmann et al.

Hoffmann et al.（2022）提出了三种估计计算最优缩放定律的方法，我们试图复制他们的第三个估计过程，其中涉及将参数损失函数拟合到从图表中重建的数据。我们发现所报道的估计与他们的前两种估计方法不一致，无法对提取的数据进行拟合，并报告了不可信的窄置信区间 - 这种窄度的区间需要超过 60 万个实验，而他们可能只进行了不到 500 个。相比之下，我们使用第三种方法重新推导缩放定律的结果与 Hoffmann 等人描述的前两种估计过程的发现是一致的。

安哥拉兔规模化实验

Chinchilla Scaling: A replication attempt

On a variety of tasks, the performance of neural networks predictably
improves with training time, dataset size and model size across many orders of
magnitude. This phenomenon is known as a neural scaling law. Of fundamental
importance is the compute-optimal scaling law, which reports the performance as
a function of units of compute when choosing model sizes optimally. We analyze
a random feature model trained with gradient descent as a solvable model of
network training and generalization. This reproduces many observations about
neural scaling laws. First, our model makes a prediction about why the scaling
of performance with training time and with model size have different power law
exponents. Consequently, the theory predicts an asymmetric compute-optimal
scaling rule where the number of training steps are increased faster than model
parameters, consistent with recent empirical observations. Second, it has been
observed that early in training, networks converge to their infinite-width
dynamics at a rate $1/\textit{width}$ but at late time exhibit a rate
$\textit{width}^{-c}$, where $c$ depends on the structure of the architecture
and task. We show that our model exhibits this behavior. Lastly, our theory
shows how the gap between training and test loss can gradually build up over
time due to repeated reuse of data.

神经网络的表现在训练时间、数据集大小和模型大小上预测性地提高，这一现象被称为神经缩放定律，而计算最优缩放定律则是将性能作为计算单元函数以选择模型大小来报告的；研究表明，神经网络在训练早期以 $1/	extit {width}$ 的速度收敛到无限宽度动力学，但在后期表现为 $	extit {width}^{-c}$ 的速度，其中 $c$ 取决于架构和任务的结构；此外，理论上显示了由于数据的重复重用，训练和测试损失之间的差距可以随时间逐渐增大。