In this work, we present a conceptually simple yet powerful baseline for the
multimodal dialog task, an S3 model, that achieves near state-of-the-art
results on two compelling leaderboards: MMMU and AI Journey Contest 2023. The
system is based on a pre-trained large language model, pre-trained modality
encoders for image and audio, and a trainable modality projector. The proposed
effective data mixture for training such an architecture demonstrates that a
multimodal model based on a strong language model and trained on a small amount
of multimodal data can perform efficiently in the task of multimodal dialog.

我们提出了一个在多模态对话任务中具有潜在简单但强大效果的基线系统 S3 模型，在 MMMU 和 AI Journey Contest 2023 两个激动人心的榜单上取得了接近最先进的成果。该系统基于一个预训练的大型语言模型，图像和音频的预训练模态编码器以及一个可训练的模态投影器。所提出的用于训练这种架构的有效数据混合表明，基于强大的语言模型并在少量多模态数据上训练的多模态模型可以高效地执行多模态对话任务。