In this study, we investigate whether the representations learned by neural networks possess a privileged and convergent basis. Specifically, we examine the significance of feature directions represented by individual neurons. First, we establish that arbitrary rotations of neural representations cannot be inverted (unlike linear networks), indicating that they do not exhibit complete rotational invariance. Subsequently, we explore the possibility of multiple bases achieving identical performance. To do this, we compare the bases of networks trained with the same parameters but with varying random initializations. Our study reveals two findings: (1) Even in wide networks such as WideResNets, neural networks do not converge to a unique basis; (2) Basis correlation increases significantly when a few early layers of the network are frozen identically. Furthermore, we analyze Linear Mode Connectivity, which has been studied as a measure of basis correlation. Our findings give evidence that while Linear Mode Connectivity improves with increased network width, this improvement is not due to an increase in basis correlation.

通过研究神经网络所学到的表示是否具有特权和趋同基础，本研究旨在探究个别神经元所表示的特征方向的重要性。结果表明，与线性网络不同，神经表示的任意旋转无法被逆转，证明其不具备完全的旋转不变性。同时，通过比较具有相同参数但具有不同随机初始化的网络的基础，我们发现：（1）即使在WideResNets等宽网络中，神经网络也无法收敛到唯一的基础；（2）当网络的少数前几层被冻结时，基础之间的相关性显著增加。此外，我们对线性模态连通性进行了分析，这是衡量基础相关性的一种方式。我们的研究发现，虽然线性模态连通性随网络宽度的增加而改善，但这种改善并不是基础相关性增加所导致的。

神经网络表示中的特权和收敛基础