Deep models are the defacto standard in visual decision problems due to their
impressive performance on a wide array of visual tasks. On the other hand,
their opaqueness has led to a surge of interest in explainable systems. In this
work, we emphasize the importance of model explanation in various forms such as
visual pointing and textual justification. The lack of data with justification
annotations is one of the bottlenecks of generating multimodal explanations.
Thus, we propose two large-scale datasets with annotations that visually and
textually justify a classification decision for various activities, i.e. ACT-X,
and for question answering, i.e. VQA-X. We also introduce a multimodal
methodology for generating visual and textual explanations simultaneously. We
quantitatively show that training with the textual explanations not only yields
better textual justification models, but also models that better localize the
evidence that support their decision.

本文介绍深度模型在视觉决策问题中表现良好的同时，其透明度也导致了可解释系统的兴起，并提出了两个带注释的大规模数据集，通过视觉和文本形式保证决策的分类，同时提出了一种多模式方法，定量证明使用文本解释的训练方法不仅可以获得更好的文本解释模型，还可以更好地定位支持其决策的证据。