While there have been many proposals on making AI algorithms explainable, few
have attempted to evaluate the impact of AI-generated explanations on human
performance in conducting human-AI collaborative tasks. To bridge the gap, we
propose a Twenty-Questions style collaborative image retrieval game,
Explanation-assisted Guess Which (ExAG), as a method of evaluating the efficacy
of explanations (visual evidence or textual justification) in the context of
Visual Question Answering (VQA). In our proposed ExAG, a human user needs to
guess a secret image picked by the VQA agent by asking natural language
questions to it. We show that overall, when AI explains its answers, users
succeed more often in guessing the secret image correctly. Notably, a few
correct explanations can readily improve human performance when VQA answers are
mostly incorrect as compared to no-explanation games. Furthermore, we also show
that while explanations rated as "helpful" significantly improve human
performance, "incorrect" and "unhelpful" explanations can degrade performance
as compared to no-explanation games. Our experiments, therefore, demonstrate
that ExAG is an effective means to evaluate the efficacy of AI-generated
explanations on a human-AI collaborative task.

通过提出 Explanation-assisted Guess Which (ExAG) 游戏作为评估 AI 生成的解释对人 - AI 协作任务产生影响的一种方法，我们发现人类在进行协作任务时，当 AI 解释其答案时，能更容易地正确猜测到秘密图像，并且有用的解释显着提高了人类的实际表现。