The second-order properties of the training loss have a massive impact on the
optimization dynamics of deep learning models. Fort & Scherlis (2019)
discovered that a high positive curvature and local convexity of the loss
Hessian are associated with highly trainable initial points located in a region
coined the "Goldilocks zone". Only a handful of subsequent studies touched upon
this relationship, so it remains largely unexplained. In this paper, we present
a rigorous and comprehensive analysis of the Goldilocks zone for homogeneous
neural networks. In particular, we derive the fundamental condition resulting
in non-zero positive curvature of the loss Hessian and argue that it is only
incidentally related to the initialization norm, contrary to prior beliefs.
Further, we relate high positive curvature to model confidence, low initial
loss, and a previously unknown type of vanishing cross-entropy loss gradient.
To understand the importance of positive curvature for trainability of deep
networks, we optimize both fully-connected and convolutional architectures
outside the Goldilocks zone and analyze the emergent behaviors. We find that
strong model performance is not necessarily aligned with the Goldilocks zone,
which questions the practical significance of this concept.

这篇论文通过对深度学习模型的优化动力学进行全面分析，探讨了训练损失的二阶性质对模型的影响，特别关注了与高度可训练的初始点相关联的 “Goldilocks zone” 概念，提出了正曲率对深度网络的可训练性的重要性，并讨论了与模型自信度、初始损失和消失的交叉熵损失梯度等方面的关联。

神经网络初始化中的金发女郎区域解构

Deconstructing the Goldilocks Zone of Neural Network Initialization

We aim to understand grokking, a phenomenon where models generalize long
after overfitting their training set. We present both a microscopic analysis
anchored by an effective theory and a macroscopic analysis of phase diagrams
describing learning performance across hyperparameters. We find that
generalization originates from structured representations whose training
dynamics and dependence on training set size can be predicted by our effective
theory in a toy setting. We observe empirically the presence of four learning
phases: comprehension, grokking, memorization, and confusion. We find
representation learning to occur only in a "Goldilocks zone" (including
comprehension and grokking) between memorization and confusion. We find on
transformers the grokking phase stays closer to the memorization phase
(compared to the comprehension phase), leading to delayed generalization. The
Goldilocks phase is reminiscent of "intelligence from starvation" in Darwinian
evolution, where resource limitations drive discovery of more efficient
solutions. This study not only provides intuitive explanations of the origin of
grokking, but also highlights the usefulness of physics-inspired tools, e.g.,
effective theories and phase diagrams, for understanding deep learning.

通过微观分析和宏观相图描述学习性能，本研究发现，泛化是由结构化表示产生的，这些表示的训练动态和对训练集大小的依赖性可以在玩具环境中通过有效理论进行预测。同时，我们观察到四个学习阶段：理解、深度理解、记忆和混淆。此外，本研究还发现，表示学习仅发生在 “Goldilocks 区域”，而这一阶段在理解和深度理解之间，而位于记忆和混淆之间。同时，我们也发现，在 transformers 模型中，深度理解阶段与记忆阶段更接近，导致延迟了泛化，这种 “Goldilocks 区域” 与达尔文进化中的 “从饥饿中获得的智慧” 类似。

理解 Grokking：表示学习有效理论的探究

Towards Understanding Grokking: An Effective Theory of Representation Learning

We explore the loss landscape of fully-connected and convolutional neural
networks using random, low-dimensional hyperplanes and hyperspheres. Evaluating
the Hessian, $H$, of the loss function on these hypersurfaces, we observe 1) an
unusual excess of the number of positive eigenvalues of $H$, and 2) a large
value of $\mathrm{Tr}(H) / ||H||$ at a well defined range of configuration
space radii, corresponding to a thick, hollow, spherical shell we refer to as
the \textit{Goldilocks zone}. We observe this effect for fully-connected neural
networks over a range of network widths and depths on MNIST and CIFAR-10
datasets with the $\mathrm{ReLU}$ and $\tanh$ non-linearities, and a similar
effect for convolutional networks. Using our observations, we demonstrate a
close connection between the Goldilocks zone, measures of local
convexity/prevalence of positive curvature, and the suitability of a network
initialization. We show that the high and stable accuracy reached when
optimizing on random, low-dimensional hypersurfaces is directly related to the
overlap between the hypersurface and the Goldilocks zone, and as a corollary
demonstrate that the notion of intrinsic dimension is initialization-dependent.
We note that common initialization techniques initialize neural networks in
this particular region of unusually high convexity/prevalence of positive
curvature, and offer a geometric intuition for their success. Furthermore, we
demonstrate that initializing a neural network at a number of points and
selecting for high measures of local convexity such as $\mathrm{Tr}(H) /
||H||$, number of positive eigenvalues of $H$, or low initial loss, leads to
statistically significantly faster training on MNIST. Based on our
observations, we hypothesize that the Goldilocks zone contains an unusually
high density of suitable initialization configurations.

通过在低维度的超平面和超球面上评估代价函数的海森矩阵，我们发现全连接神经网络和卷积神经网络的代价函数在它们的空间参数半径较大的固定范围内，具有异常的凸性和正曲率，此处我们称之为 “金发姑娘区间”，该效应与神经元网络的初始化方法，包括公共初始化技巧之间有着密切的关联。