Understanding the decisions of deep learning (DL) models is essential for the
acceptance of DL to risk-sensitive applications. Although methods, like class
activation maps (CAMs), give a glimpse into the black box, they do miss some
crucial information, thereby limiting its interpretability and merely providing
the considered locations of objects. To provide more insight into the models
and the influence of datasets, we propose an integrated feature analysis
method, which consists of feature distribution analysis and feature
decomposition, to look closer into the intermediate features extracted by DL
models. This integrated feature analysis could provide information on
overfitting, confounders, outliers in datasets, model redundancies and
principal features extracted by the models, and provide distribution
information to form a common intensity scale, which are missing in current CAM
algorithms. The integrated feature analysis was applied to eight different
datasets for general validation: photographs of handwritten digits, two
datasets of natural images and five medical datasets, including skin
photography, ultrasound, CT, X-rays and MRIs. The method was evaluated by
calculating the consistency between the CAMs average class activation levels
and the logits of the model. Based on the eight datasets, the correlation
coefficients through our method were all very close to 100%, and based on the
feature decomposition, 5%-25% of features could generate equally informative
saliency maps and obtain the same model performances as using all features.
This proves the reliability of the integrated feature analysis. As the proposed
methods rely on very few assumptions, this is a step towards better model
interpretation and a useful extension to existing CAM algorithms. Codes:
this https URL

为了更深入了解深度学习模型提取的中间特征，我们提出了一种综合特征分析方法，通过特征分布分析和特征分解来探索模型对数据集的影响，并提供了一种常用强度标尺的分布信息，以解决当前 CAM 算法中缺失的信息，验证了该方法的可靠性和相对简洁性。