The elusive nature of gradient-based optimization in neural networks is tied
to their loss landscape geometry, which is poorly understood. However recent
work has brought solid evidence that there is essentially no loss barrier
between the local solutions of gradient descent, once accounting for
weight-permutations that leave the network's computation unchanged. This raises
questions for approximate inference in Bayesian neural networks (BNNs), where
we are interested in marginalizing over multiple points in the loss landscape.
In this work, we first extend the formalism of marginalized loss barrier and
solution interpolation to BNNs, before proposing a matching algorithm to search
for linearly connected solutions. This is achieved by aligning the
distributions of two independent approximate Bayesian solutions with respect to
permutation matrices. We build on the results of Ainsworth et al. (2023),
reframing the problem as a combinatorial optimization one, using an
approximation to the sum of bilinear assignment problem. We then experiment on
a variety of architectures and datasets, finding nearly zero marginalized loss
barriers for linearly connected solutions.

通过对近似贝叶斯神经网络（BNNs）的分析，扩展了边界损失和解决方案插值的形式，并提出了一种匹配算法来寻找线性连接的解决方案。实验证明，在各种体系结构和数据集上，线性连接的解决方案几乎没有边界损失。

贝叶斯神经网络后验中的排列对称性：变分视角

On permutation symmetries in Bayesian neural network posteriors: a  variational perspective

Sharpness-aware minimization (SAM), which searches for flat minima by min-max
optimization, has been shown to be useful in improving model generalization.
However, since each SAM update requires computing two gradients, its
computational cost and training time are both doubled compared to standard
empirical risk minimization (ERM). Recent state-of-the-arts reduce the fraction
of SAM updates and thus accelerate SAM by switching between SAM and ERM updates
randomly or periodically. In this paper, we design an adaptive policy to employ
SAM based on the loss landscape geometry. Two efficient algorithms, AE-SAM and
AE-LookSAM, are proposed. We theoretically show that AE-SAM has the same
convergence rate as SAM. Experimental results on various datasets and
architectures demonstrate the efficiency and effectiveness of the adaptive
policy.

本文提出了一种自适应策略 AE-SAM 和 AE-LookSAM 来精准控制模型使用 sharpness-aware minimization（SAM）。经实验证明该方案具有较高效率和效果。

利用锐度敏感的最小化策略

An Adaptive Policy to Employ Sharpness-Aware Minimization

Linear interpolation between initial neural network parameters and converged
parameters after training with stochastic gradient descent (SGD) typically
leads to a monotonic decrease in the training objective. This Monotonic Linear
Interpolation (MLI) property, first observed by Goodfellow et al. (2014)
persists in spite of the non-convex objectives and highly non-linear training
dynamics of neural networks. Extending this work, we evaluate several
hypotheses for this property that, to our knowledge, have not yet been
explored. Using tools from differential geometry, we draw connections between
the interpolated paths in function space and the monotonicity of the network -
providing sufficient conditions for the MLI property under mean squared error.
While the MLI property holds under various settings (e.g. network architectures
and learning problems), we show in practice that networks violating the MLI
property can be produced systematically, by encouraging the weights to move far
from initialization. The MLI property raises important questions about the loss
landscape geometry of neural networks and highlights the need to further study
their global properties.

通过分析神经网络的初始参数和训练后的参数之间的线性插值关系，本文首次给出了神经网络训练中的单调线性插值（MLI）现象，通过利用微分几何理论，提供了 MSE 中 MLI 出现的充分条件，并发现了当网络权重远离初始化时，将出现违反 MLI 现象的情况。