When speakers describe an image, they tend to look at objects before
mentioning them. In this paper, we investigate such sequential cross-modal
alignment by modelling the image description generation process
computationally. We take as our starting point a state-of-the-art image
captioning system and develop several model variants that exploit information
from human gaze patterns recorded during language production. In particular, we
propose the first approach to image description generation where visual
processing is modelled $\textit{sequentially}$. Our experiments and analyses
confirm that better descriptions can be obtained by exploiting gaze-driven
attention and shed light on human cognitive processes by comparing different
ways of aligning the gaze modality with language production. We find that
processing gaze data sequentially leads to descriptions that are better aligned
to those produced by speakers, more diverse, and more natural${-}$particularly
when gaze is encoded with a dedicated recurrent component.

本文通过计算机模拟探索图像描述生成时的跨模态对齐过程，结合人类凝视模式记录的信息进行多种模型方案的开发并分析，提出在视觉处理中逐序建模的图像描述生成方法，研究发现使凝视数据逐序处理可以得到更好的描述，探讨了人类认知过程的不同对齐方式并取得了更多自然、多样和更贴近说话者生成描述的结果。