In this paper, we propose to guide the video caption generation with
Part-of-Speech (POS) information, based on a gated fusion of multiple
representations of input videos. We construct a novel gated fusion network,
with one particularly designed cross-gating (CG) block, to effectively encode
and fuse different types of representations, e.g., the motion and content
features of an input video. One POS sequence generator relies on this fused
representation to predict the global syntactic structure, which is thereafter
leveraged to guide the video captioning generation and control the syntax of
the generated sentence. Specifically, a gating strategy is proposed to
dynamically and adaptively incorporate the global syntactic POS information
into the decoder for generating each word. Experimental results on two
benchmark datasets, namely MSR-VTT and MSVD, demonstrate that the proposed
model can well exploit complementary information from multiple representations,
resulting in improved performances. Moreover, the generated global POS
information can well capture the global syntactic structure of the sentence,
and thus be exploited to control the syntactic structure of the description.
Such POS information not only boosts the video captioning performance but also
improves the diversity of the generated captions. Our code is at:
this https URL

本文提出了一种基于 Part-of-Speech 信息引导视频字幕生成的方法，通过多个输入视频表示的门控融合，构建了一种新颖的门控融合网络，其中一个特别设计的交叉门控块可以有效地编码和融合不同类型的表示，结果显示该模型可充分利用多种表示的互补信息，提高了性能。