Recent works have proposed that activations in language models can be
modelled as sparse linear combinations of vectors corresponding to features of
input text. Under this assumption, these works aimed to reconstruct feature
directions using sparse coding. We develop metrics to assess the success of
these sparse coding techniques and test the validity of the linearity and
sparsity assumptions. We show our metrics can predict the level of sparsity on
synthetic sparse linear activations, and can distinguish between sparse linear
data and several other distributions. We use our metrics to measure levels of
sparsity in several language models. We find evidence that language model
activations can be accurately modelled by sparse linear combinations of
features, significantly more so than control datasets. We also show that model
activations appear to be sparsest in the first and final layers.

最近的研究工作提出了一种假设，即语言模型中的激活可以被建模为对应于输入文本特征的向量的稀疏线性组合。在这个假设下，这些工作旨在使用稀疏编码重构特征方向。我们开发了度量方法来评估这些稀疏编码技术的成功，并测试线性和稀疏假设的有效性。我们展示了我们的度量方法可以预测合成稀疏线性激活的稀疏水平，并可以区分稀疏线性数据和其他几种分布。我们使用我们的度量方法来测量几个语言模型中的稀疏水平。我们发现有证据表明语言模型的激活可以准确地被特征的稀疏线性组合所建模，这种情况比控制数据集要显著得多。我们还展示了模型的激活在第一层和最后一层似乎是最稀疏的。

量化语言模型中的特征稀疏性

Measuring Feature Sparsity in Language Models

In this study, we investigate whether the representations learned by neural
networks possess a privileged and convergent basis. Specifically, we examine
the significance of feature directions represented by individual neurons.
First, we establish that arbitrary rotations of neural representations cannot
be inverted (unlike linear networks), indicating that they do not exhibit
complete rotational invariance. Subsequently, we explore the possibility of
multiple bases achieving identical performance. To do this, we compare the
bases of networks trained with the same parameters but with varying random
initializations. Our study reveals two findings: (1) Even in wide networks such
as WideResNets, neural networks do not converge to a unique basis; (2) Basis
correlation increases significantly when a few early layers of the network are
frozen identically.
Furthermore, we analyze Linear Mode Connectivity, which has been studied as a
measure of basis correlation. Our findings give evidence that while Linear Mode
Connectivity improves with increased network width, this improvement is not due
to an increase in basis correlation.

通过研究神经网络所学到的表示是否具有特权和趋同基础，本研究旨在探究个别神经元所表示的特征方向的重要性。结果表明，与线性网络不同，神经表示的任意旋转无法被逆转，证明其不具备完全的旋转不变性。同时，通过比较具有相同参数但具有不同随机初始化的网络的基础，我们发现：（1）即使在 WideResNets 等宽网络中，神经网络也无法收敛到唯一的基础；（2）当网络的少数前几层被冻结时，基础之间的相关性显著增加。此外，我们对线性模态连通性进行了分析，这是衡量基础相关性的一种方式。我们的研究发现，虽然线性模态连通性随网络宽度的增加而改善，但这种改善并不是基础相关性增加所导致的。