The development of language models have moved from encoder-decoder to
decoder-only designs. In addition, the common knowledge has it that the two
most popular multimodal tasks, the generative and contrastive tasks, tend to
conflict with one another, are hard to accommodate in one architecture, and
further need complex adaptations for downstream tasks. We propose a novel
paradigm of training with a decoder-only model for multimodal tasks, which is
surprisingly effective in jointly learning of these disparate vision-language
tasks. This is done with a simple model, called MaMMUT. It consists of a single
vision encoder and a text decoder, and is able to accommodate contrastive and
generative learning by a novel two-pass approach on the text decoder. We
demonstrate that joint training of these diverse-objective tasks is simple,
effective, and maximizes the weight-sharing of the model. Furthermore, the same
architecture enables straightforward extensions to open-vocabulary object
detection and video-language tasks. The model tackles a diverse range of tasks,
while being modest in capacity. Our model achieves the SOTA on image-text and
text-image retrieval, video question answering and open-vocabulary detection
tasks, outperforming much larger and more extensively trained foundational
models. It shows competitive results on VQA and Video Captioning, especially
considering its size. Ablations confirm the flexibility and advantages of our
approach.

我们提出了一种使用解码器模型进行多模式任务训练的新范例，其中 MaMMUT 作为一个简单的模型，能够通过新颖的文本解码器的两次传递方法容纳对比和生成学习，并能够直接扩展到开放词汇的对象检测和视频语言任务，且该模型在多个任务上均取得了最佳效果。