Existing methods for vision-and-language learning typically require designing task-specific architectures and objectives for each task. For example, a multi-label answer classifier for visual question answering, a region scorer for referring expression comprehension, and a language decoder for image captioning, etc. To alleviate these hassles, in this work, we propose a unified framework that learns different tasks in a single architecture with the same language modeling objective, i.e., multimodal conditional text generation, where our models learn to generate labels in text based on the visual and textual inputs. On 7 popular vision-and-language benchmarks, including visual question answering, referring expression comprehension, visual commonsense reasoning, most of which have been previously modeled as discriminative tasks, our generative approach (with a single unified architecture) reaches comparable performance to recent task-specific state-of-the-art vision-and-language models. Moreover, our generative approach shows better generalization ability on answering questions that have rare answers. In addition, we show that our framework allows multi-task learning in a single architecture with a single set of parameters, which achieves similar performance to separately optimized single-task models. Our code will be publicly available at: https://github.com/j-min/VL-T5

该研究提出了一个统一框架，通过相同的语言建模目标，在单个体系结构中学习不同的任务，实现视频图像理解、语言推理和标签生成等领域的应用。通过在多个基准测试中的表现，这种生成方法（在单个统一的体系结构下）显示出了与最先进的特定任务模型相当的性能，并且还显示了更好的推广能力和单个任务模型相似的多任务学习能力。

通过文本生成统一视觉语言任务