The field of vision and language has witnessed a proliferation of pre-trained
foundation models. Most existing methods are independently pre-trained with
contrastive objective like CLIP, image-to-text generative objective like PaLI,
or text-to-image generative objective like Parti. However, the three objectives
can be pre-trained on the same data, image-text pairs, and intuitively they
complement each other as contrasting provides global alignment capacity and
generation grants fine-grained understanding. In this work, we present a
Contrastive Bi-directional Image-Text generation model (CoBIT), which attempts
to unify the three pre-training objectives in one framework. Specifically,
CoBIT employs a novel unicoder-decoder structure, consisting of an image
unicoder, a text unicoder and a cross-modal decoder. The image/text unicoders
can switch between encoding and decoding in different tasks, enabling
flexibility and shared knowledge that benefits both image-to-text and
text-to-image generations. CoBIT achieves superior performance in image
understanding, image-text understanding (Retrieval, Captioning, VQA, SNLI-VE)
and text-based content creation, particularly in zero-shot scenarios. For
instance, 82.7% in zero-shot ImageNet classification, 9.37 FID score in
zero-shot text-to-image generation and 44.8 CIDEr in zero-shot captioning.

提出了一个新的 Contrastive Bi-directional Image-Text generation (CoBIT) 模型，具有 unicoder-decoder 结构，可同时应用于图像理解、图像与文本相互理解和文本创作，特别是在零样本场景下表现卓越。