Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world. However, it has been observed that large-scale pretraining usually can result in better generalization performance, e.g., CLIP (Contrastive Language-Image Pre-training), trained on a massive amount of image-caption pairs, has shown a strong zero-shot capability on various vision tasks. To further study the advantage brought by CLIP, we propose to use CLIP as the visual encoder in various V&L models in two typical scenarios: 1) plugging CLIP into task-specific fine-tuning; 2) combining CLIP with V&L pre-training and transferring to downstream tasks. We show that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data, such as BottomUp-TopDown. We achieve competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering, Visual Entailment, and V&L Navigation tasks. We release our code at https://github.com/clip-vil/CLIP-ViL.

研究了在视觉与语言任务中使用大规模预训练模型CLIP作为视觉编码器以及其优势，通过在特定任务中微调和在预训练模型中与V&L相结合传递到下游任务，CLIP显著优于现有的视觉编码器，并在多种视觉与语言任务中取得竞争或更好的结果，同时取得了Visual Question Answering，Visual Entailment和V&L Navigation等任务的新高峰。

CLIP能为视觉语言任务带来多大的效益？