Artificial intelligence (AI) has huge potential to improve the health and well-being of people, but adoption in clinical practice is still limited. Lack of transparency is identified as one of the main barriers to implementation, as clinicians should be confident the AI system can be trusted. Explainable AI has the potential to overcome this issue and can be a step towards trustworthy AI. In this paper we review the recent literature to provide guidance to researchers and practitioners on the design of explainable AI systems for the health-care domain and contribute to formalization of the field of explainable AI. We argue the reason to demand explainability determines what should be explained as this determines the relative importance of the properties of explainability (i.e. interpretability and fidelity). Based on this, we give concrete recommendations to choose between classes of explainable AI methods (explainable modelling versus post-hoc explanation; model-based, attribution-based, or example-based explanations; global and local explanations). Furthermore, we find that quantitative evaluation metrics, which are important for objective standardized evaluation, are still lacking for some properties (e.g. clarity) and types of explanators (e.g. example-based methods). We conclude that explainable modelling can contribute to trustworthy AI, but recognize that complementary measures might be needed to create trustworthy AI (e.g. reporting data quality, performing extensive (external) validation, and regulation).

本文回顾了近期文献，提出了可解释人工智能系统的设计指南以及可解释人工智能领域的规范化贡献。该文章提出了一个框架，以指导选择可解释人工智能方法的类别，进而解释重要属性的相对重要性。同时也发现，尽管定量评价指标对于客观标准化评估很重要，但对于某些属性（如“清晰度”）和解释类型（如“基于实例的方法”）仍然缺乏。最终的结论是，可解释建模可以为可信的人工智能做出贡献，但还需要在实践中证明可解释性的好处，并可能需要采取补充措施，例如报告数据质量，进行广泛（外部）验证和监管等措施。

The role of explainability in creating trustworthy artificial
  intelligence for health care: a comprehensive survey of the terminology,
  design choices, and evaluation strategies

可解释性在构建值得信赖的医疗人工智能中的作用：术语、设计选择和评估策略的综合调查