This tutorial note summarizes the presentation on ``Large Multimodal Models:
Towards Building and Surpassing Multimodal GPT-4'', a part of CVPR 2023
tutorial on ``Recent Advances in Vision Foundation Models''. The tutorial
consists of three parts. We first introduce the background on recent GPT-like
large models for vision-and-language modeling to motivate the research in
instruction-tuned large multimodal models (LMMs). As a pre-requisite, we
describe the basics of instruction-tuning in large language models, which is
further extended to the multimodal space. Lastly, we illustrate how to build
the minimum prototype of multimodal GPT-4 like models with the open-source
resource, and review the recently emerged topics.

该论文总结了 CVPR2023 最新视觉基础模型进展讲座中有关多模式 GPT-4 模型的演示，介绍了最新的大规模语言模型中的指令调整、多模态空间的扩展，以及如何用开源资源构建多模态 GPT-4 模型的最小原型等热门话题。

大型多模态模型：CVPR 2023 教程笔记

Large Multimodal Models: Notes on CVPR 2023 Tutorial

Recent advances in vision-and-language modeling have seen the development of
Transformer architectures that achieve remarkable performance on multimodal
reasoning tasks. Yet, the exact capabilities of these black-box models are
still poorly understood. While much of previous work has focused on studying
their ability to learn meaning at the word-level, their ability to track
syntactic dependencies between words has received less attention. We take a
first step in closing this gap by creating a new multimodal task targeted at
evaluating understanding of predicate-noun dependencies in a controlled setup.
We evaluate a range of state-of-the-art models and find that their performance
on the task varies considerably, with some models performing relatively well
and others at chance level. In an effort to explain this variability, our
analyses indicate that the quality (and not only sheer quantity) of pretraining
data is essential. Additionally, the best performing models leverage
fine-grained multimodal pretraining objectives in addition to the standard
image-text matching objectives. This study highlights that targeted and
controlled evaluations are a crucial step for a precise and rigorous test of
the multimodal knowledge of vision-and-language models.

本文研究视觉 - 语言建模，通过创建新的多模态任务和分析预训练数据的质量，发现预训练数据的质量和多模态预训练目标对模型的性能影响重要。