This paper presents OmniVL, a new foundation model to support both
image-language and video-language tasks using one universal architecture. It
adopts a unified transformer-based visual encoder for both image and video
inputs, and thus can perform joint image-language and video-language
pretraining. We demonstrate, for the first time, such a paradigm benefits both
image and video tasks, as opposed to the conventional one-directional transfer
(e.g., use image-language to help video-language). To this end, we propose a
decoupled joint pretraining of image-language and video-language to effectively
decompose the vision-language modeling into spatial and temporal dimensions and
obtain performance boost on both image and video tasks. Moreover, we introduce
a novel unified vision-language contrastive (UniVLC) loss to leverage
image-text, video-text, image-label (e.g., image classification), video-label
(e.g., video action recognition) data together, so that both supervised and
noisily supervised pretraining data are utilized as much as possible. Without
incurring extra task-specific adaptors, OmniVL can simultaneously support
visual only tasks (e.g., image classification, video action recognition),
cross-modal alignment tasks (e.g., image/video-text retrieval), and multi-modal
understanding and generation tasks (e.g., image/video question answering,
captioning). We evaluate OmniVL on a wide range of downstream tasks and achieve
state-of-the-art or competitive results with similar model size and data scale.

OmniVL 提出了一种新的基础模型，它通过使用一种通用架构来支持图像语言和视频语言任务，采用了一种统一的基于 Transformer 的视觉编码器，以此来实现联合图像语言和视频语言的预训练，并展示了这种范式的好处，并且这种模型能够同时支持视觉任务，跨模态任务和多模态理解。