We present a method to improve video description generation by modeling
higher-order interactions between video frames and described concepts. By
storing past visual attention in the video associated to previously generated
words, the system is able to decide what to look at and describe in light of
what it has already looked at and described. This enables not only more
effective local attention, but tractable consideration of the video sequence
while generating each word. Evaluation on the challenging and popular MSVD and
Charades datasets demonstrates that the proposed architecture outperforms
previous video description approaches without requiring external temporal video
features.

本文提出了一种通过建模视频帧和描述概念之间的高阶交互来改善视频描述生成的方法。通过存储先前与之关联的视觉注意力，系统能够决定在已经看过和描述过的内容的基础上看什么并进行描述。这不仅可以实现更有效的局部关注，而且在生成每个单词时可以实现可处理的视频序列的考虑。在具有挑战性和广受欢迎的 MSVD 和 Charades 数据集上的评估表明，所提出的体系结构优于以前的视频描述方法，而无需外部时间视频特征。