We introduce a unified framework to jointly model images, text, and human
attention traces. Our work is built on top of the recent Localized Narratives
annotation framework [30], where each word of a given caption is paired with a
mouse trace segment. We propose two novel tasks: (1) predict a trace given an
image and caption (i.e., visual grounding), and (2) predict a caption and a
trace given only an image. Learning the grounding of each word is challenging,
due to noise in the human-provided traces and the presence of words that cannot
be meaningfully visually grounded. We present a novel model architecture that
is jointly trained on dual tasks (controlled trace generation and controlled
caption generation). To evaluate the quality of the generated traces, we
propose a local bipartite matching (LBM) distance metric which allows the
comparison of two traces of different lengths. Extensive experiments show our
model is robust to the imperfect training data and outperforms the baselines by
a clear margin. Moreover, we demonstrate that our model pre-trained on the
proposed tasks can be also beneficial to the downstream task of COCO's guided
image captioning. Our code and project page are publicly available.

本文提出了一个统一的框架来综合地模拟图像、文本和人类注意力轨迹，并设计了两个任务，即视觉基础任务和图像生成任务。该模型架构同时训练于两个任务上，使用了本文设计的局部二分匹配距离度量方法来评估生成的轨迹的质量，并在完整实验中得出了模型优于基线的结论。此外，我们还证明了预训练的模型对 COCO 的引导图像字幕的下游任务也有积极的效果。