Artificial Intelligence in Medicine has made significant progress with emerging applications in medical imaging, patient care, and other areas. While these applications have proven successful in retrospective studies, very few of them were applied in practice.The field of Medical AI faces various challenges, in terms of building user trust, complying with regulations, using data ethically.Explainable AI (XAI) aims to enable humans understand AI and trust its results. This paper presents a literature review on the recent developments of XAI solutions for medical decision support, based on a representative sample of 198 articles published in recent years. The systematic synthesis of the relevant articles resulted in several findings. (1) model-agnostic XAI techniques were mostly employed in these solutions, (2) deep learning models are utilized more than other types of machine learning models, (3) explainability was applied to promote trust, but very few works reported the physicians participation in the loop, (4) visual and interactive user interface is more useful in understanding the explanation and the recommendation of the system. More research is needed in collaboration between medical and AI experts, that could guide the development of suitable frameworks for the design, implementation, and evaluation of XAI solutions in medicine.

近年来，医疗人工智能领域已取得显著进展，但面临着建立用户信任、符合法规和合乎道德使用数据等挑战。这篇论文针对代表性样本中的198篇文章对医疗决策支持的可解释人工智能解决方案进行了文献综述，发现大多数解决方案采用了模型无关的可解释人工智能技术，深度学习模型被广泛应用，用户界面的可视化和交互更有助于理解解释和系统推荐。在医疗和人工智能专家之间加强合作，以指导可解释人工智能解决方案在医学领域的设计、实施和评估，仍需要开展更多的研究。

医疗领域中的可解释人工智能应用：系统综述