The energy landscape of high-dimensional non-convex optimization problems is
crucial to understanding the effectiveness of modern deep neural network
architectures. Recent works have experimentally shown that two different
solutions found after two runs of a stochastic training are often connected by
very simple continuous paths (e.g., linear) modulo a permutation of the
weights. In this paper, we provide a framework theoretically explaining this
empirical observation. Based on convergence rates in Wasserstein distance of
empirical measures, we show that, with high probability, two wide enough
two-layer neural networks trained with stochastic gradient descent are linearly
connected. Additionally, we express upper and lower bounds on the width of each
layer of two deep neural networks with independent neuron weights to be
linearly connected. Finally, we empirically demonstrate the validity of our
approach by showing how the dimension of the support of the weight distribution
of neurons, which dictates Wasserstein convergence rates is correlated with
linear mode connectivity.

理论上解释了以往实验观察到的两次随机训练之后找到的两个不同解经常通过简单连续路径（例如线性）通过权重的排列变换相连的现象，基于经验测度的 Wasserstein 距离的收敛速度，我们证明了用随机梯度下降训练的足够宽的两层神经网络的线性连接性，并且对具有独立神经元权重的两个深度神经网络的每层宽度的上下界给出了线性连接性，最后通过展示权重分布支持的维度与线性模式连接性的相关性来验证我们方法的有效性。