A key to deciphering the inner workings of neural networks is understanding
what a model has learned. Promising methods for discovering learned features
are based on analyzing activation values, whereby current techniques focus on
analyzing high activation values to reveal interesting features on a neuron
level. However, analyzing high activation values limits layer-level concept
discovery. We present a method that instead takes into account the entire
activation distribution. By extracting similar activation profiles within the
high-dimensional activation space of a neural network layer, we find groups of
inputs that are treated similarly. These input groups represent neural
activation patterns (NAPs) and can be used to visualize and interpret learned
layer concepts. We release a framework with which NAPs can be extracted from
pre-trained models and provide a visual introspection tool that can be used to
analyze NAPs. We tested our method with a variety of networks and show how it
complements existing methods for analyzing neural network activation values.

通过提取神经网络层中高维激活空间内相似的激活特点，我们找到了一些被对待相似的输入群体，这些输入群体代表神经激活模式 (NAPs)，并可用于可视化和解释学习层概念，从而实现了神经网络的分析方法的互补。