Deep neural networks learn increasingly complex functions over the course of
training. Here, we show both empirically and theoretically that learning of the
target function is preceded by an early phase in which networks learn the
optimal constant solution (OCS) - that is, initial model responses mirror the
distribution of target labels, while entirely ignoring information provided in
the input. Using a hierarchical category learning task, we derive exact
solutions for learning dynamics in deep linear networks trained with bias
terms. Even when initialized to zero, this simple architectural feature induces
substantial changes in early dynamics. We identify hallmarks of this early OCS
phase and illustrate how these signatures are observed in deep linear networks
and larger, more complex (and nonlinear) convolutional neural networks solving
a hierarchical learning task based on MNIST and CIFAR10. We explain these
observations by proving that deep linear networks necessarily learn the OCS
during early learning. To further probe the generality of our results, we train
human learners over the course of three days on the category learning task. We
then identify qualitative signatures of this early OCS phase in terms of the
dynamics of true negative (correct-rejection) rates. Surprisingly, we find the
same early reliance on the OCS in the behaviour of human learners. Finally, we
show that learning of the OCS can emerge even in the absence of bias terms and
is equivalently driven by generic correlations in the input data. Overall, our
work suggests the OCS as a universal learning principle in supervised,
error-corrective learning, and the mechanistic reasons for its prevalence.

在这篇研究中，我们展示了实证和理论上的结果，证明深度神经网络在学习目标函数之前存在一个早期阶段，其中网络学习最佳常数解（OCS），即初始模型的响应与目标标签的分布相匹配，完全忽略输入中提供的信息。通过使用分层类别学习任务，我们推导出具有偏移项训练的深度线性网络学习动力学的精确解。即使初始化为零，这个简单的结构特征也会引起早期动力学的显著变化。我们确定了这个早期 OCS 阶段的特征，并说明了这些特征在深度线性网络和更复杂（非线性）卷积神经网络在基于 MNIST 和 CIFAR10 的分层学习任务中的观察结果。我们通过证明深度线性网络在早期学习过程中必然学习 OCS 来解释这些观察结果。为了进一步探索我们结果的普遍性，我们训练人类学习者在三天的课程中进行类别学习任务。然后，我们通过真实负样本（正确拒绝）率的动态性质确定了这个早期 OCS 阶段的定性特征。令人惊讶的是，我们发现人类学习者的行为中也存在对 OCS 的早期依赖。最后，我们证明 OCS 的学习甚至可以在没有偏移项的情况下出现，并且同样受到输入数据中的通用相关性的驱动。总体上，我们的研究表明 OCS 是一种在监督式纠错学习中普遍存在的学习原则，并阐明了其普遍性的机制原因。

神经网络和人类中的最优恒定解的早期学习

Early learning of the optimal constant solution in neural networks and  humans

Conventional wisdom suggests that neural network predictions tend to be
unpredictable and overconfident when faced with out-of-distribution (OOD)
inputs. Our work reassesses this assumption for neural networks with
high-dimensional inputs. Rather than extrapolating in arbitrary ways, we
observe that neural network predictions often tend towards a constant value as
input data becomes increasingly OOD. Moreover, we find that this value often
closely approximates the optimal constant solution (OCS), i.e., the prediction
that minimizes the average loss over the training data without observing the
input. We present results showing this phenomenon across 8 datasets with
different distributional shifts (including CIFAR10-C and ImageNet-R, S),
different loss functions (cross entropy, MSE, and Gaussian NLL), and different
architectures (CNNs and transformers). Furthermore, we present an explanation
for this behavior, which we first validate empirically and then study
theoretically in a simplified setting involving deep homogeneous networks with
ReLU activations. Finally, we show how one can leverage our insights in
practice to enable risk-sensitive decision-making in the presence of OOD
inputs.

神经网络预测在面对越来越多的分布外数据时，往往呈现出不可预测和过度自信的特点。然而，我们的研究发现，与其任意拟合，神经网络预测往往趋向于一个恒定值，并且这个值通常接近于最优恒定解（OCS），即在没有观察到输入时最小化训练数据平均损失的预测。我们在 8 个包含不同分布转移的数据集（包括 CIFAR10-C 和 ImageNet-R,S）、不同损失函数（交叉熵、均方误差和高斯负对数似然）以及不同架构（CNN 和转换器）上展示了这种现象。此外，我们提出了这种行为的解释，首先通过实证研究验证，然后在简化的深度均质网络和 ReLU 激活的环境中进行理论研究。最后，我们展示了如何在面对分布外输入时利用我们的发现在实践中实现风险敏感的决策。