Single neurons in neural networks are often ``interpretable'' in that they
represent individual, intuitively meaningful features. However, many neurons
exhibit $\textit{mixed selectivity}$, i.e., they represent multiple unrelated
features. A recent hypothesis proposes that features in deep networks may be
represented in $\textit{superposition}$, i.e., on non-orthogonal axes by
multiple neurons, since the number of possible interpretable features in
natural data is generally larger than the number of neurons in a given network.
Accordingly, we should be able to find meaningful directions in activation
space that are not aligned with individual neurons. Here, we propose (1) an
automated method for quantifying visual interpretability that is validated
against a large database of human psychophysics judgments of neuron
interpretability, and (2) an approach for finding meaningful directions in
network activation space. We leverage these methods to discover directions in
convolutional neural networks that are more intuitively meaningful than
individual neurons, as we confirm and investigate in a series of analyses.
Moreover, we apply the same method to two recent datasets of visual neural
responses in the brain and find that our conclusions largely transfer to real
neural data, suggesting that superposition might be deployed by the brain. This
also provides a link with disentanglement and raises fundamental questions
about robust, efficient and factorized representations in both artificial and
biological neural systems.

在神经网络中，单个神经元往往是 “可解释的”，因为它们代表了个别直观有意义的特征。然而，许多神经元表现出混合选择性，即它们代表多个不相关的特征。最近的假设提出深度网络中的特征可能通过多个神经元以非正交的方式进行 “叠加”，因为自然数据中可解释的特征数量通常大于给定网络中的神经元数量。因此，我们应该能够在激活空间中找到与个别神经元不一致的有意义方向。在这里，我们提出了（1）一种自动化方法，用于量化视觉可解释性，并与人类心理物理学对神经元可解释性的大型数据库验证，以及（2）一种在网络激活空间中找到有意义方向的方法。我们利用这些方法在卷积神经网络中发现了比个别神经元更具直观意义的方向，通过一系列分析进行了确认和调查。此外，我们将相同的方法应用于大脑中两个最近的视觉神经响应数据集，并发现我们的结论在大部分转移到真实神经数据上，这表明大脑可能使用了叠加。这也与去纠缠有关，并在人工和生物神经系统中提出了关于稳健、高效和分解表示的基本问题。

在人工和生物神经系统中识别可解释的视觉特征

Identifying Interpretable Visual Features in Artificial and Biological  Neural Systems

Neural networks can implement arbitrary functions. But, mechanistically, what
are the tools at their disposal to construct the target? For classification
tasks, the network must transform the data classes into a linearly separable
representation in the final hidden layer. We show that a feed-forward
architecture has one primary tool at hand to achieve this separability:
progressive folding of the data manifold in unoccupied higher dimensions. The
operation of folding provides a useful intuition in low-dimensions that
generalizes to high ones. We argue that an alternative method based on shear,
requiring very deep architectures, plays only a small role in real-world
networks. The folding operation, however, is powerful as long as layers are
wider than the data dimensionality, allowing efficient solutions by providing
access to arbitrary regions in the distribution, such as data points of one
class forming islands within the other classes. We argue that a link exists
between the universal approximation property in ReLU networks and the
fold-and-cut theorem (Demaine et al., 1998) dealing with physical paper
folding. Based on the mechanistic insight, we predict that the progressive
generation of separability is necessarily accompanied by neurons showing mixed
selectivity and bimodal tuning curves. This is validated in a network trained
on the poker hand task, showing the emergence of bimodal tuning curves during
training. We hope that our intuitive picture of the data transformation in deep
networks can help to provide interpretability, and discuss possible
applications to the theory of convolutional networks, loss landscapes, and
generalization.
TL;DR: Shows that the internal processing of deep networks can be thought of
as literal folding operations on the data distribution in the N-dimensional
activation space. A link to a well-known theorem in origami theory is provided.

通过将数据类折叠到未被占用的高维度中，神经网络的内部处理可以被看作是对 N 维度激活空间中的数据分布进行文字操作。带有 Origami 理论中著名定理的链接。