The ability to explain decisions made by AI systems is highly sought after,
especially in domains where human lives are at stake such as medicine or
autonomous vehicles. While it is often possible to approximate the input-output
relations of deep neural networks with a few human-understandable rules, the
discovery of the double descent phenomena suggests that such approximations do
not accurately capture the mechanism by which deep neural networks work. Double
descent indicates that deep neural networks typically operate by smoothly
interpolating between data points rather than by extracting a few high level
rules. As a result, neural networks trained on complex real world data are
inherently hard to interpret and prone to failure if asked to extrapolate. To
show how we might be able to trust AI despite these problems we introduce the
concept of self-explaining AI. Self-explaining AIs are capable of providing a
human-understandable explanation of each decision along with confidence levels
for both the decision and explanation. For this approach to work, it is
important that the explanation actually be related to the decision, ideally
capturing the mechanism used to arrive at the explanation. Finally, we argue it
is important that deep learning based systems include a "warning light" based
on techniques from applicability domain analysis to warn the user if a model is
asked to extrapolate outside its training distribution. For a video
presentation of this talk see this https URL& .

本研究提出了一种自我解释的 AI 的概念，它能够解释其决策的机制并提供信心水平，但由于深度神经网络的双下降现象，训练复杂真实世界数据的神经网络难以解释和外推，因此深度学习系统需要包括特定预警来保证其应用的安全性。