Recent interpretability methods propose using concept-based explanations to
translate the internal representations of deep learning models into a language
that humans are familiar with: concepts. This requires understanding which
concepts are present in the representation space of a neural network. One
popular method for finding concepts is Concept Activation Vectors (CAVs), which
are learnt using a probe dataset of concept exemplars. In this work, we
investigate three properties of CAVs. CAVs may be: (1) inconsistent between
layers, (2) entangled with different concepts, and (3) spatially dependent.
Each property provides both challenges and opportunities in interpreting
models. We introduce tools designed to detect the presence of these properties,
provide insight into how they affect the derived explanations, and provide
recommendations to minimise their impact. Understanding these properties can be
used to our advantage. For example, we introduce spatially dependent CAVs to
test if a model is translation invariant with respect to a specific concept and
class. Our experiments are performed on ImageNet and a new synthetic dataset,
Elements. Elements is designed to capture a known ground truth relationship
between concepts and classes. We release this dataset to facilitate further
research in understanding and evaluating interpretability methods.

使用概念解释将深度学习模型的内部表示转化为人类熟悉的语言，最近的解释性方法提议采用基于概念的解释。本文研究了 Concept Activation Vectors (CAVs) 的三个性质，它们可能在不同层次之间存在不一致性，与不同概念纠缠在一起，并具有空间依赖性，这些性质为解释模型提供了挑战和机会。同时，我们还介绍了工具来检测这些性质的存在，并提供洞见以了解它们对解释的影响，并提出减小它们影响的建议。通过理解这些性质，我们可以利用它们的优势。例如，我们引入了具有空间依赖性的 CAVs 来测试模型在特定概念和类别上是否具有平移不变性。我们在 ImageNet 和一个新的合成数据集 Elements 上进行实验。Elements 旨在捕捉概念与类别之间已知的真实关系。我们发布此数据集以促进对解释性方法的进一步研究和评估。