Exploring the loss landscape offers insights into the inherent principles of deep neural networks (DNNs). Recent work suggests an additional asymmetry of the valley beyond the flat and sharp ones, yet without thoroughly examining its causes or implications. Our study methodically explores the factors affecting the symmetry of DNN valleys, encompassing (1) the dataset, network architecture, initialization, and hyperparameters that influence the convergence point; and (2) the magnitude and direction of the noise for 1D visualization. Our major observation shows that the {\it degree of sign consistency} between the noise and the convergence point is a critical indicator of valley symmetry. Theoretical insights from the aspects of ReLU activation and softmax function could explain the interesting phenomenon. Our discovery propels novel understanding and applications in the scenario of Model Fusion: (1) the efficacy of interpolating separate models significantly correlates with their sign consistency ratio, and (2) imposing sign alignment during federated learning emerges as an innovative approach for model parameter alignment.

探索深度神经网络（DNNs）的损失景观可以揭示其内在原理。我们的研究系统地探索了影响DNNs山谷对称性的因素，包括数据集、网络架构、初始化、超参数以及噪声的大小和方向。我们的主要观察表明，噪声与收敛点之间的符号一致性是山谷对称性的关键指标。从ReLU激活函数和softmax函数的角度可以解释这一有趣现象。我们的发现在模型融合方案中带来新的理解和应用，特别是插值单独模型的有效性与它们的符号一致性比例相关，并在联邦学习中实施符号对齐成为模型参数对齐的创新方法。

深度神经网络的非对称谷探索与利用