The opacity of deep neural networks remains a challenge in deploying
solutions where explanation is as important as precision. We present ConceptX,
a human-in-the-loop framework for interpreting and annotating latent
representational space in pre-trained Language Models (pLMs). We use an
unsupervised method to discover concepts learned in these models and enable a
graphical interface for humans to generate explanations for the concepts. To
facilitate the process, we provide auto-annotations of the concepts (based on
traditional linguistic ontologies). Such annotations enable development of a
linguistic resource that directly represents latent concepts learned within
deep NLP models. These include not just traditional linguistic concepts, but
also task-specific or sensitive concepts (words grouped based on gender or
religious connotation) that helps the annotators to mark bias in the model. The
framework consists of two parts (i) concept discovery and (ii) annotation
platform.

该研究提出了 ConceptX 人在循环框架，用于解释深度语言模型中预训练隐藏表达空间，并提供自动生成的基于语言本体论的概念注释，以表示这些模型中学习到的隐含概念，从而帮助注释人员标记模型中的偏见。

ConceptX：潜在概念分析框架

ConceptX: A Framework for Latent Concept Analysis

We propose a novel framework ConceptX, to analyze how latent concepts are
encoded in representations learned within pre-trained language models. It uses
clustering to discover the encoded concepts and explains them by aligning with
a large set of human-defined concepts. Our analysis on seven transformer
language models reveal interesting insights: i) the latent space within the
learned representations overlap with different linguistic concepts to a varying
degree, ii) the lower layers in the model are dominated by lexical concepts
(e.g., affixation), whereas the core-linguistic concepts (e.g., morphological
or syntactic relations) are better represented in the middle and higher layers,
iii) some encoded concepts are multi-faceted and cannot be adequately explained
using the existing human-defined concepts.

本文提出了一个新颖的框架 ConceptX，利用聚类发现预训练语言模型中编码的潜在概念，并通过与大量人类定义的概念进行对齐进行解释。它在七个变压器语言模型上的分析揭示了有趣的见解：i）学习表示中的潜在空间以不同的程度与不同的语言概念重叠，ii）模型中的较低层由词汇概念（例如，词缀）主导，而核心语言概念（例如，形态或句法关系）在中高层中更好地表示，iii）一些编码的概念具有多面性，无法用现有的人类定义概念充分说明。