We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a question. Thus, we introduce the VQA-HAT (Human ATtention) dataset. We evaluate attention maps generated by state-of-the-art VQA models against human attention both qualitatively (via visualizations) and quantitatively (via rank-order correlation). We find that depending on the implementation used, machine-generated attention maps are either \emph{negatively correlated} with human attention or have positive correlation worse than task-independent saliency. Overall, our experiments paint a bleak picture for the current generation of attention models in VQA.

我们通过设计游戏化的注意力注释接口，在Visual Question Answering（VQA）中对人类注意力进行了大规模研究，为此引入了VQA-HAT数据集，在定性和定量方面评估VQA模型生成的注意力信息与人类注意力的相似性。研究结果表明，当前的VQA注意力模型似乎没有关注与人类注意力相同的区域。

视觉问答中的人类关注：人类和深度神经网络是否看同一区域？