Explainable deep learning models are advantageous in many situations. Prior
work mostly provide unimodal explanations through post-hoc approaches not part
of the original system design. Explanation mechanisms also ignore useful
textual information present in images. In this paper, we propose MTXNet, an
end-to-end trainable multimodal architecture to generate multimodal
explanations, which focuses on the text in the image. We curate a novel dataset
TextVQA-X, containing ground truth visual and multi-reference textual
explanations that can be leveraged during both training and evaluation. We then
quantitatively show that training with multimodal explanations complements
model performance and surpasses unimodal baselines by up to 7% in CIDEr scores
and 2% in IoU. More importantly, we demonstrate that the multimodal
explanations are consistent with human interpretations, help justify the
models' decision, and provide useful insights to help diagnose an incorrect
prediction. Finally, we describe a real-world e-commerce application for using
the generated multimodal explanations.

该论文提出了一种可解释的深度学习模型 - MTXNet，着重于图像中的文本的多模态解释生成，并使用包含视觉信息和多种文本解释的新颖数据集 TextVQA-X 进行训练和评估，显示出该多模态解释与人类解释一致，有助于为模型的决策提供有用的见解。该模型解释效果的得分在 CIDEr 方面优于单模态模型至多 7％，在 IoU 方面优于单模态模型至多 2％。