Explanations are hypothesized to improve human understanding of machine
learning models and achieve a variety of desirable outcomes, ranging from model
debugging to enhancing human decision making. However, empirical studies have
found mixed and even negative results. An open question, therefore, is under
what conditions explanations can improve human understanding and in what way.
Using adapted causal diagrams, we provide a formal characterization of the
interplay between machine explanations and human understanding, and show how
human intuitions play a central role in enabling human understanding.
Specifically, we identify three core concepts of interest that cover all
existing quantitative measures of understanding in the context of human-AI
decision making: task decision boundary, model decision boundary, and model
error. Our key result is that without assumptions about task-specific
intuitions, explanations may potentially improve human understanding of model
decision boundary, but they cannot improve human understanding of task decision
boundary or model error. To achieve complementary human-AI performance, we
articulate possible ways on how explanations need to work with human
intuitions. For instance, human intuitions about the relevance of features
(e.g., education is more important than age in predicting a person's income)
can be critical in detecting model error. We validate the importance of human
intuitions in shaping the outcome of machine explanations with empirical
human-subject studies. Overall, our work provides a general framework along
with actionable implications for future algorithmic development and empirical
experiments of machine explanations.

本研究通过适应性因果图的方式，探讨了人机共存下机器学习模型解释与人类理解的相互作用。研究发现，提高人类针对模型决策边界的理解是可能的，但提高人类对任务决策边界或模型错误的理解需要结合人类主观性经验去实现。最后，本研究提出了一些可行的措施，以及未来机器学习模型解释研究的方向。