Vision-language pre-training and instruction tuning have demonstrated
general-purpose capabilities in 2D visual reasoning tasks by aligning visual
encoders with state-of-the-art large language models (LLMs). In this paper, we
introduce a simple, yet effective, cross-modality framework built atop frozen
LLMs that allows the integration of various modalities without extensive
modality-specific customization. To facilitate instruction-modality
fine-tuning, we collect high-quality instruction tuning data in an automatic
and scalable manner, composed of 24K QA samples for audio and 250K QA samples
for 3D. Leveraging instruction-aware representations, our model performs
comparably with leading-edge counterparts without the need of extensive
modality-specific pre-training or customization. Furthermore, our approach
demonstrates cross-modal reasoning abilities across two or more input
modalities, despite each modality projection being trained individually. To
study the model's cross-modal abilities, we contribute a novel Discriminative
Cross-modal Reasoning (DisCRn) evaluation task, comprising 9K audio-video QA
samples and 28K image-3D QA samples that require the model to reason
discriminatively across disparate input modalities.

我们在这篇论文中介绍了一个简单而有效的跨模态框架，利用现有的大型语言模型，在 2D 视觉推理任务中通过将视觉编码器与最先进的语言模型对齐，展现了通用功能。我们通过收集高质量的调整数据，自动和可扩展地收集音频和 3D 方面 QA 样本，以便实现指导模态的微调。通过利用指导感知表达，我们的模型在无需大规模特定模态的预训练或自定义的情况下，表现出与领先模型相当的性能。此外，我们的方法展现出在两个或更多输入模态中进行跨模态推理的能力，尽管每个模态投影都是单独训练的。为了研究模型的跨模态能力，我们提出了一个新颖的鉴别跨模态推理（DisCRn）评估任务，包括 9K 个音频 - 视频 QA 样本和 28K 个图像 - 3D QA 样本，要求模型在不同的输入模态之间进行鉴别性推理。