Understanding intermediate representations of the concepts learned by deep
learning classifiers is indispensable for interpreting general model behaviors.
Existing approaches to reveal learned concepts often rely on human supervision,
such as pre-defined concept sets or segmentation processes. In this paper, we
propose a novel unsupervised method for discovering distributed representations
of concepts by selecting a principal subset of neurons. Our empirical findings
demonstrate that instances with similar neuron activation states tend to share
coherent concepts. Based on the observations, the proposed method selects
principal neurons that construct an interpretable region, namely a Relaxed
Decision Region (RDR), encompassing instances with coherent concepts in the
feature space. It can be utilized to identify unlabeled subclasses within data
and to detect the causes of misclassifications. Furthermore, the applicability
of our method across various layers discloses distinct distributed
representations over the layers, which provides deeper insights into the
internal mechanisms of the deep learning model.

本文提出了一种新的无监督方法，通过选择主要的神经元子集来发现概念的分布式表示，该方法可以用于识别数据中的未标记子类，并检测错误分类的原因，同时还揭示了不同层次的深度学习模型内部机制的深入洞察。

无监督条件下理解深度神经网络中概念的分布表示

Understanding Distributed Representations of Concepts in Deep Neural  Networks without Supervision

The out-of-distribution (OOD) problem generally arises when neural networks
encounter data that significantly deviates from the training data distribution,
\ie, in-distribution (InD). In this paper, we study the OOD problem from a
neuron activation view. We first formulate neuron activation states by
considering both the neuron output and its influence on model decisions. Then,
we propose the concept of \textit{neuron activation coverage} (NAC), which
characterizes the neuron behaviors under InD and OOD data. Leveraging our NAC,
we show that 1) InD and OOD inputs can be naturally separated based on the
neuron behavior, which significantly eases the OOD detection problem and
achieves a record-breaking performance of 0.03% FPR95 on ResNet-50,
outperforming the previous best method by 20.67%; 2) a positive correlation
between NAC and model generalization ability consistently holds across
architectures and datasets, which enables a NAC-based criterion for evaluating
model robustness. By comparison with the traditional validation criterion, we
show that NAC-based criterion not only can select more robust models, but also
has a stronger correlation with OOD test performance.

本文提出了一种神经元激活覆盖度（NAC）概念，并通过研究神经元行为以区分 InD 和 OOD 数据，打破了 ResNet-50 FPR95 记录并提出了 NAC 标准以评估模型健壮性。