Video-and-language pre-training has shown promising improvements on various
downstream tasks. Most previous methods capture cross-modal interactions with a
transformer-based multimodal encoder, not fully addressing the misalignment
between unimodal video and text features. Besides, learning fine-grained
visual-language alignment usually requires off-the-shelf object detectors to
provide object information, which is bottlenecked by the detector's limited
vocabulary and expensive computation cost.
We propose Align and Prompt: an efficient and effective video-and-language
pre-training framework with better cross-modal alignment. First, we introduce a
video-text contrastive (VTC) loss to align unimodal video-text features at the
instance level, which eases the modeling of cross-modal interactions. Then, we
propose a new visually-grounded pre-training task, prompting entity modeling
(PEM), which aims to learn fine-grained region-entity alignment. To achieve
this, we first introduce an entity prompter module, which is trained with VTC
to produce the similarity between a video crop and text prompts instantiated
with entity names. The PEM task then asks the model to predict the entity
pseudo-labels (i.e~normalized similarity scores) for randomly-selected video
crops. The resulting pre-trained model achieves state-of-the-art performance on
both text-video retrieval and videoQA, outperforming prior work by a
substantial margin. Our code and pre-trained models are available at
this https URL

本文提出了一种高效且有效的视频和语言预训练框架 ——Align and Prompt，通过提出视频 - 文本对比（VTC）损失和提示实体建模（PEM）任务来更好地跨模态对齐，从而学习精细的区域 - 实体对齐，与以前的方法相比取得了显着性能提升。