Three important criteria of existing convolutional neural networks (CNNs) are
(1) test-set accuracy; (2) out-of-distribution accuracy; and (3)
explainability. While these criteria have been studied independently, their
relationship is unknown. For example, do CNNs that have a stronger
out-of-distribution performance have also stronger explainability? Furthermore,
most prior feature-importance studies only evaluate methods on 2-3 common
vanilla ImageNet-trained CNNs, leaving it unknown how these methods generalize
to CNNs of other architectures and training algorithms. Here, we perform the
first, large-scale evaluation of the relations of the three criteria using 9
feature-importance methods and 12 ImageNet-trained CNNs that are of 3 training
algorithms and 5 CNN architectures. We find several important insights and
recommendations for ML practitioners. First, adversarially robust CNNs have a
higher explainability score on gradient-based attribution methods (but not
CAM-based or perturbation-based methods). Second, AdvProp models, despite being
highly accurate more than both vanilla and robust models alone, are not
superior in explainability. Third, among 9 feature attribution methods tested,
GradCAM and RISE are consistently the best methods. Fourth, Insertion and
Deletion are biased towards vanilla and robust models respectively, due to
their strong correlation with the confidence score distributions of a CNN.
Fifth, we did not find a single CNN to be the best in all three criteria, which
interestingly suggests that CNNs are harder to interpret as they become more
accurate.

本文使用 9 种特征重要性方法和 12 种各种架构和培训算法的 ImageNet CNN 进行了第一次大规模评估，并发现了其中的一些重要洞察和建议，例如，GradCAM 和 RISE 是最佳方法，插入和删除分别对应于普通和稳健模型。