The attention mechanisms in deep neural networks are inspired by human's
attention that sequentially focuses on the most relevant parts of the
information over time to generate prediction output. The attention parameters
in those models are implicitly trained in an end-to-end manner, yet there have
been few trials to explicitly incorporate human gaze tracking to supervise the
attention models. In this paper, we investigate whether attention models can
benefit from explicit human gaze labels, especially for the task of video
captioning. We collect a new dataset called VAS, consisting of movie clips, and
corresponding multiple descriptive sentences along with human gaze tracking
data. We propose a video captioning model named Gaze Encoding Attention Network
(GEAN) that can leverage gaze tracking information to provide the spatial and
temporal attention for sentence generation. Through evaluation of language
similarity metrics and human assessment via Amazon mechanical Turk, we
demonstrate that spatial attentions guided by human gaze data indeed improve
the performance of multiple captioning methods. Moreover, we show that the
proposed approach achieves the state-of-the-art performance for both gaze
prediction and video captioning not only in our VAS dataset but also in
standard datasets (e.g. LSMDC and Hollywood2).

本研究探讨是否可以利用人眼注视追踪信息定义注意力机制以提高视频字幕生成任务的性能，通过提出一种名为 GEAN 的视频字幕模型，并采用人眼注视追踪数据来提供生成句子的时空注意力，以及对语言相似性指标和人工智能通过 Amazon mechanical Turk 进行的评估，证明了由人眼追踪数据指导的空间注意力确实改善了多个字幕方法的性能，并展示了该方法在 VAS 数据集和标准数据集（如 LSMDC 和 Hollywood2）中实现了领先的性能，成为最先进的视频字幕生成方法。