Artificial Intelligence in Medicine has made significant progress with
emerging applications in medical imaging, patient care, and other areas. While
these applications have proven successful in retrospective studies, very few of
them were applied in practice.The field of Medical AI faces various challenges,
in terms of building user trust, complying with regulations, using data
ethically.Explainable AI (XAI) aims to enable humans understand AI and trust
its results. This paper presents a literature review on the recent developments
of XAI solutions for medical decision support, based on a representative sample
of 198 articles published in recent years. The systematic synthesis of the
relevant articles resulted in several findings. (1) model-agnostic XAI
techniques were mostly employed in these solutions, (2) deep learning models
are utilized more than other types of machine learning models, (3)
explainability was applied to promote trust, but very few works reported the
physicians participation in the loop, (4) visual and interactive user interface
is more useful in understanding the explanation and the recommendation of the
system. More research is needed in collaboration between medical and AI
experts, that could guide the development of suitable frameworks for the
design, implementation, and evaluation of XAI solutions in medicine.

近年来，医疗人工智能领域已取得显著进展，但面临着建立用户信任、符合法规和合乎道德使用数据等挑战。这篇论文针对代表性样本中的 198 篇文章对医疗决策支持的可解释人工智能解决方案进行了文献综述，发现大多数解决方案采用了模型无关的可解释人工智能技术，深度学习模型被广泛应用，用户界面的可视化和交互更有助于理解解释和系统推荐。在医疗和人工智能专家之间加强合作，以指导可解释人工智能解决方案在医学领域的设计、实施和评估，仍需要开展更多的研究。