Being able to explain the prediction to clinical end-users is a necessity to
leverage the power of artificial intelligence (AI) models for clinical decision
support. For medical images, a feature attribution map, or heatmap, is the most
common form of explanation that highlights important features for AI models'
prediction. However, it is unknown how well heatmaps perform on explaining
decisions on multi-modal medical images, where each image modality or channel
visualizes distinct clinical information of the same underlying biomedical
phenomenon. Understanding such modality-dependent features is essential for
clinical users' interpretation of AI decisions. To tackle this clinically
important but technically ignored problem, we propose the modality-specific
feature importance (MSFI) metric. It encodes clinical image and explanation
interpretation patterns of modality prioritization and modality-specific
feature localization. We conduct a clinical requirement-grounded, systematic
evaluation using computational methods and a clinician user study. Results show
that the examined 16 heatmap algorithms failed to fulfill clinical requirements
to correctly indicate AI model decision process or decision quality. The
evaluation and MSFI metric can guide the design and selection of XAI algorithms
to meet clinical requirements on multi-modal explanation.

本论文针对医学图像 AI 模型预测结果的解释问题，提出了基于不同模态的特征重要性度量方法，对 16 种 heatmap 算法进行评估，结果表明这些算法不能很好地解释多模态医学图像的 AI 模型预测结果，因此需要基于该研究提出的度量方法设计 XAI 算法以满足临床应用要求。