The development of language models have moved from encoder-decoder to decoder-only designs. In addition, the common knowledge has it that the two most popular multimodal tasks, the generative and contrastive tasks, tend to conflict with one another, are hard to accommodate in one architecture, and further need complex adaptations for downstream tasks. We propose a novel paradigm of training with a decoder-only model for multimodal tasks, which is surprisingly effective in jointly learning of these disparate vision-language tasks. This is done with a simple model, called MaMMUT. It consists of a single vision encoder and a text decoder, and is able to accommodate contrastive and generative learning by a novel two-pass approach on the text decoder. We demonstrate that joint training of these diverse-objective tasks is simple, effective, and maximizes the weight-sharing of the model. Furthermore, the same architecture enables straightforward extensions to open-vocabulary object detection and video-language tasks. The model tackles a diverse range of tasks, while being modest in capacity. Our model achieves the SOTA on image-text and text-image retrieval, video question answering and open-vocabulary detection tasks, outperforming much larger and more extensively trained foundational models. It shows competitive results on VQA and Video Captioning, especially considering its size. Ablations confirm the flexibility and advantages of our approach.

我们提出了一种使用解码器模型进行多模式任务训练的新范例，其中MaMMUT作为一个简单的模型，能够通过新颖的文本解码器的两次传递方法容纳对比和生成学习，并能够直接扩展到开放词汇的对象检测和视频语言任务，且该模型在多个任务上均取得了最佳效果。

MaMMUT：联合学习多模态任务的简单架构