The recent large-scale vision-language pre-training (VLP) of dual-stream
architectures (e.g., CLIP) with a tremendous amount of image-text pair data,
has shown its superiority on various multimodal alignment tasks. Despite its
success, the resulting models are not capable of multimodal generative tasks
due to the weak text encoder. To tackle this problem, we propose to augment the
dual-stream VLP model with a textual pre-trained language model (PLM) via
vision-language knowledge distillation (VLKD), enabling the capability for
multimodal generation. VLKD is pretty data- and computation-efficient compared
to the pre-training from scratch. Experimental results show that the resulting
model has strong zero-shot performance on multimodal generation tasks, such as
open-ended visual question answering and image captioning. For example, it
achieves 44.5% zero-shot accuracy on the VQAv2 dataset, surpassing the previous
state-of-the-art zero-shot model with $7\times$ fewer parameters. Furthermore,
the original textual language understanding and generation ability of the PLM
is maintained after VLKD, which makes our model versatile for both multimodal
and unimodal tasks.

通过视觉 - 语言知识蒸馏 (VLKD) 增强双流 VLP 模型，使其具有多模态生成能力，实现开放式视觉问答和图像字幕等多模态生成任务的强零 - shot 性能。