Pretrained language models (PLMs) form the basis of most state-of-the-art NLP
technologies. Nevertheless, they are essentially black boxes: Humans do not
have a clear understanding of what knowledge is encoded in different parts of
the models, especially in individual neurons. The situation is different in
computer vision, where feature visualization provides a decompositional
interpretability technique for neurons of vision models. Activation
maximization is used to synthesize inherently interpretable visual
representations of the information encoded in individual neurons. Our work is
inspired by this but presents a cautionary tale on the interpretability of
single neurons, based on the first large-scale attempt to adapt activation
maximization to NLP, and, more specifically, large PLMs. We propose feature
textualization, a technique to produce dense representations of neurons in the
PLM word embedding space. We apply feature textualization to the BERT model
(Devlin et al., 2019) to investigate whether the knowledge encoded in
individual neurons can be interpreted and symbolized. We find that the produced
representations can provide insights about the knowledge encoded in individual
neurons, but that individual neurons do not represent clearcut symbolic units
of language such as words. Additionally, we use feature textualization to
investigate how many neurons are needed to encode words in BERT.

计算机视觉中可视化特征提供了一种将视觉模型的神经元信息分解解释的方法，我们的研究受到这一方法的启发并针对大型预训练语言模型进行了活化最大化的尝试；我们提出了特征文本化技术，用于提取预训练语言模型中神经元的知识表示，发现这些表示可以揭示关于神经元所编码的知识的见解，但神经元并不代表清晰的语言符号单元如词。