Interpreting deep neural networks is the topic of much current research in
AI. However, few interpretability techniques have shown to be competitive tools
in practical applications. Inspired by how benchmarks tend to guide progress in
AI, we make three contributions. First, we propose trojan rediscovery as a
benchmarking task to evaluate how useful interpretability tools are for
generating engineering-relevant insights. Second, we design two such approaches
for benchmarking: one for feature attribution methods and one for feature
synthesis methods. Third, we apply our benchmarks to evaluate 16 feature
attribution/saliency methods and 9 feature synthesis methods. This approach
finds large differences in the capabilities of these existing tools and shows
significant room for improvement. Finally, we propose several directions for
future work. Resources are available at
this https URL

本文提出 Trojan rediscovery 作为一项基准测试任务，以评估可解释性工具生成工程相关见解的实用价值，并设计了两种基准测试方法。最终将这些方法应用到 16 个特征归因 / 显著性方法和 9 个特征合成方法中，发现了现有工具之间的巨大差异，并指出了未来研究的方向。