The inductive bias of a neural network is largely determined by the
architecture and the training algorithm. To achieve good generalization, how to
effectively train a neural network is of great importance. We propose a novel
orthogonal over-parameterized training (OPT) framework that can provably
minimize the hyperspherical energy which characterizes the diversity of neurons
on a hypersphere. By maintaining the minimum hyperspherical energy during
training, OPT can greatly improve the empirical generalization. Specifically,
OPT fixes the randomly initialized weights of the neurons and learns an
orthogonal transformation that applies to these neurons. We consider multiple
ways to learn such an orthogonal transformation, including unrolling
orthogonalization algorithms, applying orthogonal parameterization, and
designing orthogonality-preserving gradient descent. For better scalability, we
propose the stochastic OPT which performs orthogonal transformation
stochastically for partial dimensions of neurons. Interestingly, OPT reveals
that learning a proper coordinate system for neurons is crucial to
generalization. We provide some insights on why OPT yields better
generalization. Extensive experiments validate the superiority of OPT over the
standard training.

该研究提出了一种新的正交超参数训练 (OPT) 框架，该框架可以证明地最小化描述神经元多样性的超球面能量，通过保持最小超球面能量来提高神经网络的泛化能力。

正交过参数化训练

Orthogonal Over-Parameterized Training

Inspired by the Thomson problem in physics where the distribution of multiple
propelling electrons on a unit sphere can be modeled via minimizing some
potential energy, hyperspherical energy minimization has demonstrated its
potential in regularizing neural networks and improving their generalization
power. In this paper, we first study the important role that hyperspherical
energy plays in neural network training by analyzing its training dynamics.
Then we show that naively minimizing hyperspherical energy suffers from some
difficulties due to highly non-linear and non-convex optimization as the space
dimensionality becomes higher, therefore limiting the potential to further
improve the generalization. To address these problems, we propose the
compressive minimum hyperspherical energy (CoMHE) as a more effective
regularization for neural networks. Specifically, CoMHE utilizes projection
mappings to reduce the dimensionality of neurons and minimizes their
hyperspherical energy. According to different designs for the projection
mapping, we propose several distinct yet well-performing variants and provide
some theoretical guarantees to justify their effectiveness. Our experiments
show that CoMHE consistently outperforms existing regularization methods, and
can be easily applied to different neural networks.

探究后者如何通过对神经网络训练的分析来理解去偏好能量曲面，而提出了投影映射以减少神经元维度并最小化其超球能量的压缩最小超球能量作为神经网络的更有效的正则化方法。