Deep Neural Networks (DNNs) draw their power from the representations they
learn. However, while being incredibly effective in learning complex
abstractions, they are susceptible to learning malicious concepts, due to the
spurious correlations inherent in the training data. So far, existing methods
for uncovering such artifactual behavior in trained models focus on finding
artifacts in the input data, which requires both availability of a data set and
human supervision. In this paper, we introduce DORA (Data-agnOstic
Representation Analysis): the first data-agnostic framework for the analysis of
the representation space of DNNs. We propose a novel distance measure between
representations that utilizes self-explaining capabilities within the network
itself without access to any data and quantitatively validate its alignment
with human-defined semantic distances. We further demonstrate that this metric
could be utilized for the detection of anomalous representations, which may
bear a risk of learning unintended spurious concepts deviating from the desired
decision-making policy. Finally, we demonstrate the practical utility of DORA
by analyzing and identifying artifactual representations in widely popular
Computer Vision models.

这篇论文介绍了 DORA（Data-agnOstic Representation Analysis）框架，它利用网络内部的自我解释能力，提出了一种新的距离度量方法，对 DNNs 的表示空间进行了数据无关的分析。通过这个度量方法，可以检测到可能存在风险的异常表示，该方法最终应用于分析并识别了广受欢迎的计算机视觉模型中的人工异常表示。