There often is a dilemma between ease of optimization and robust
out-of-distribution (OoD) generalization. For instance, many OoD methods rely
on penalty terms whose optimization is challenging. They are either too strong
to optimize reliably or too weak to achieve their goals.
We propose to initialize the networks with a rich representation containing a
palette of potentially useful features, ready to be used by even simple models.
On the one hand, a rich representation provides a good initialization for the
optimizer. On the other hand, it also provides an inductive bias that helps OoD
generalization. Such a representation is constructed with the Rich Feature
Construction (RFC) algorithm, also called the Bonsai algorithm, which consists
of a succession of training episodes. During discovery episodes, we craft a
multi-objective optimization criterion and its associated datasets in a manner
that prevents the network from using the features constructed in the previous
iterations. During synthesis episodes, we use knowledge distillation to force
the network to simultaneously represent all the previously discovered features.
Initializing the networks with Bonsai representations consistently helps six
OoD methods achieve top performance on ColoredMNIST benchmark. The same
technique substantially outperforms comparable results on the Wilds Camelyon17
task, eliminates the high result variance that plagues other methods, and makes
hyperparameter tuning and model selection more reliable.

该研究针对模型优化和鲁棒性之间的矛盾，提出了一种用于网络初始化的丰富特征构建算法，并证明了该算法有助于实现 OoD 方法的最佳性能。

优化与泛化困境的丰富特征构建

Rich Feature Construction for the Optimization-Generalization Dilemma

Learning visual features from unlabeled image data is an important yet
challenging task, which is often achieved by training a model on some
annotation-free information. We consider spatial contexts, for which we solve
so-called jigsaw puzzles, i.e., each image is cut into grids and then
disordered, and the goal is to recover the correct configuration. Existing
approaches formulated it as a classification task by defining a fixed mapping
from a small subset of configurations to a class set, but these approaches
ignore the underlying relationship between different configurations and also
limit their application to more complex scenarios. This paper presents a novel
approach which applies to jigsaw puzzles with an arbitrary grid size and
dimensionality. We provide a fundamental and generalized principle, that weaker
cues are easier to be learned in an unsupervised manner and also transfer
better. In the context of puzzle recognition, we use an iterative manner which,
instead of solving the puzzle all at once, adjusts the order of the patches in
each step until convergence. In each step, we combine both unary and binary
features on each patch into a cost function judging the correctness of the
current configuration. Our approach, by taking similarity between puzzles into
consideration, enjoys a more reasonable way of learning visual knowledge. We
verify the effectiveness of our approach in two aspects. First, it is able to
solve arbitrarily complex puzzles, including high-dimensional puzzles, that
prior methods are difficult to handle. Second, it serves as a reliable way of
network initialization, which leads to better transfer performance in a few
visual recognition tasks including image classification, object detection, and
semantic segmentation.

本文提出了一种用于解决任意大小和形状的智力拼图，利用无监督学习从未标记的图像数据中学习视觉特征，可作为网络初始化的可靠方法，适用于图像分类、物体检测和语义分割等几类视觉识别任务。

弱空间约束下的迭代重组：无人监督表征学习中任意拼图问题的解决

Iterative Reorganization with Weak Spatial Constraints: Solving  Arbitrary Jigsaw Puzzles for Unsupervised Representation Learning

We explore the loss landscape of fully-connected and convolutional neural
networks using random, low-dimensional hyperplanes and hyperspheres. Evaluating
the Hessian, $H$, of the loss function on these hypersurfaces, we observe 1) an
unusual excess of the number of positive eigenvalues of $H$, and 2) a large
value of $\mathrm{Tr}(H) / ||H||$ at a well defined range of configuration
space radii, corresponding to a thick, hollow, spherical shell we refer to as
the \textit{Goldilocks zone}. We observe this effect for fully-connected neural
networks over a range of network widths and depths on MNIST and CIFAR-10
datasets with the $\mathrm{ReLU}$ and $\tanh$ non-linearities, and a similar
effect for convolutional networks. Using our observations, we demonstrate a
close connection between the Goldilocks zone, measures of local
convexity/prevalence of positive curvature, and the suitability of a network
initialization. We show that the high and stable accuracy reached when
optimizing on random, low-dimensional hypersurfaces is directly related to the
overlap between the hypersurface and the Goldilocks zone, and as a corollary
demonstrate that the notion of intrinsic dimension is initialization-dependent.
We note that common initialization techniques initialize neural networks in
this particular region of unusually high convexity/prevalence of positive
curvature, and offer a geometric intuition for their success. Furthermore, we
demonstrate that initializing a neural network at a number of points and
selecting for high measures of local convexity such as $\mathrm{Tr}(H) /
||H||$, number of positive eigenvalues of $H$, or low initial loss, leads to
statistically significantly faster training on MNIST. Based on our
observations, we hypothesize that the Goldilocks zone contains an unusually
high density of suitable initialization configurations.

通过在低维度的超平面和超球面上评估代价函数的海森矩阵，我们发现全连接神经网络和卷积神经网络的代价函数在它们的空间参数半径较大的固定范围内，具有异常的凸性和正曲率，此处我们称之为 “金发姑娘区间”，该效应与神经元网络的初始化方法，包括公共初始化技巧之间有着密切的关联。