Interpretable deep learning models have received widespread attention in the
field of image recognition. Due to the unique multi-instance learning of
medical images and the difficulty in identifying decision-making regions, many
interpretability models that have been proposed still have problems of
insufficient accuracy and interpretability in medical image disease diagnosis.
To solve these problems, we propose feature-driven inference network
(FeaInfNet). Our first key innovation involves proposing a feature-based
network reasoning structure, which is applied to FeaInfNet. The network of this
structure compares the similarity of each sub-region image patch with the
disease templates and normal templates that may appear in the region, and
finally combines the comparison of each sub-region to make the final diagnosis.
It simulates the diagnosis process of doctors to make the model interpretable
in the reasoning process, while avoiding the misleading caused by the
participation of normal areas in reasoning. Secondly, we propose local feature
masks (LFM) to extract feature vectors in order to provide global information
for these vectors, thus enhancing the expressive ability of the FeaInfNet.
Finally, we propose adaptive dynamic masks (Adaptive-DM) to interpret feature
vectors and prototypes into human-understandable image patches to provide
accurate visual interpretation. We conducted qualitative and quantitative
experiments on multiple publicly available medical datasets, including RSNA,
iChallenge-PM, Covid-19, ChinaCXRSet, and MontgomerySet. The results of our
experiments validate that our method achieves state-of-the-art performance in
terms of classification accuracy and interpretability compared to baseline
methods in medical image diagnosis. Additional ablation studies verify the
effectiveness of each of our proposed components.

通过提出特征驱动的推理网络（FeaInfNet），本研究解决了医学图像疾病诊断中可解释性模型的问题，并通过实验证实其在多个公开医学数据集上与基准方法相比在分类准确性和可解释性方面达到了最先进的性能。

FeaInfNet: 基于特征驱动推理和可视化解释的医学图像诊断

FeaInfNet: Diagnosis in Medical Image with Feature-Driven Inference and  Visual Explanations

The design of interpretable deep learning models working in relational
domains poses an open challenge: interpretable deep learning methods, such as
Concept-Based Models (CBMs), are not designed to solve relational problems,
while relational models are not as interpretable as CBMs. To address this
problem, we propose Relational Concept-Based Models, a family of relational
deep learning methods providing interpretable task predictions. Our
experiments, ranging from image classification to link prediction in knowledge
graphs, show that relational CBMs (i) match generalization performance of
existing relational black-boxes (as opposed to non-relational CBMs), (ii)
support the generation of quantified concept-based explanations, (iii)
effectively respond to test-time interventions, and (iv) withstand demanding
settings including out-of-distribution scenarios, limited training data
regimes, and scarce concept supervisions.

我们提出了一种解决关系领域中可解释深度学习模型设计问题的方法：关系概念模型（Relational Concept-Based Models）。实验结果表明，关系概念模型与现有的关系黑匣子模型在泛化性能上相匹配，并支持生成定量概念解释，有效地响应测试时干预，并适应多种严苛环境，包括分布不均、训练数据有限和概念监督不足。

基于关系概念的模型

Relational Concept Based Models

Deep learning (DL) models have been popular due to their ability to learn
directly from the raw data in an end-to-end paradigm, alleviating the concern
of a separate error-prone feature extraction phase. Recent DL-based
neuroimaging studies have also witnessed a noticeable performance advancement
over traditional machine learning algorithms. But the challenges of deep
learning models still exist because of the lack of transparency in these models
for their successful deployment in real-world applications. In recent years,
Explainable AI (XAI) has undergone a surge of developments mainly to get
intuitions of how the models reached the decisions, which is essential for
safety-critical domains such as healthcare, finance, and law enforcement
agencies. While the interpretability domain is advancing noticeably,
researchers are still unclear about what aspect of model learning a post hoc
method reveals and how to validate its reliability. This paper comprehensively
reviews interpretable deep learning models in the neuroimaging domain. Firstly,
we summarize the current status of interpretability resources in general,
focusing on the progression of methods, associated challenges, and opinions.
Secondly, we discuss how multiple recent neuroimaging studies leveraged model
interpretability to capture anatomical and functional brain alterations most
relevant to model predictions. Finally, we discuss the limitations of the
current practices and offer some valuable insights and guidance on how we can
steer our future research directions to make deep learning models substantially
interpretable and thus advance scientific understanding of brain disorders.

本文综述了神经影像领域中可解释深度学习模型的当前状态和应用，讨论了其相关方法、挑战和意见，以及如何利用模型解释性捕捉与模型预测相关的解剖和功能性大脑变化，最后讨论了当前做法的局限性，并提供了有关如何引导未来研究方向以使深度学习模型更加可解释并推进对脑疾病科学理解的宝贵见解和指导。