Multimodal headline utilizes both video frames and transcripts to generate
the natural language title of the videos. Due to a lack of large-scale,
manually annotated data, the task of annotating grounded headlines for video is
labor intensive and impractical. Previous researches on pre-trained language
models and video-language models have achieved significant progress in related
downstream tasks. However, none of them can be directly applied to multimodal
headline architecture where we need both multimodal encoder and sentence
decoder. A major challenge in simply gluing language model and video-language
model is the modality balance, which is aimed at combining visual-language
complementary abilities. In this paper, we propose a novel approach to graft
the video encoder from the pre-trained video-language model on the generative
pre-trained language model. We also present a consensus fusion mechanism for
the integration of different components, via inter/intra modality relation.
Empirically, experiments show that the grafted model achieves strong results on
a brand-new dataset collected from real-world applications.

本文提出了一种新方法，将预训练的视频 - 语言模型中的视频编码器嫁接到生成式的预训练语言模型上，并通过跨模态和内模态关系的一致性融合机制集成不同组件，以实现多模标题的生成。经过实验证明，嫁接模型在从实际应用程序收集的全新数据集上取得了强大的结果。