Recent video and language pretraining frameworks lack the ability to generate
sentences. We present Multimodal Video Generative Pretraining (MV-GPT), a new
pretraining framework for learning from unlabelled videos which can be
effectively used for generative tasks such as multimodal video captioning.
Unlike recent video-language pretraining frameworks, our framework trains both
a multimodal video encoder and a sentence decoder jointly. To overcome the lack
of captions in unlabelled videos, we leverage the future utterance as an
additional text source and propose a bidirectional generation objective -- we
generate future utterances given the present mulitmodal context, and also the
present utterance given future observations. With this objective, we train an
encoder-decoder model end-to-end to generate a caption from raw pixels and
transcribed speech directly. Our model achieves state-of-the-art performance
for multimodal video captioning on four standard benchmarks, as well as for
other video understanding tasks such as VideoQA, video retrieval and action
classification.

提出了一种新的预训练框架 Multimodal Video Generative Pretraining (MV-GPT)，通过利用未标记视频中的未来话语作为附加文本源并引入双向生成目标，以从生图像和录制语音直接生成说明的端到端模型来有效地生成多模态视频说明。