Instruction-tuned large language models (LLMs) have demonstrated promising
zero-shot generalization capabilities across various downstream tasks. Recent
research has introduced multimodal capabilities to LLMs by integrating
independently pretrained vision encoders through model grafting. These
multimodal variants undergo instruction tuning, similar to LLMs, enabling
effective zero-shot generalization for multimodal tasks. This study conducts a
comparative analysis of different multimodal instruction tuning approaches and
evaluates their performance across a range of tasks, including complex
reasoning, conversation, image captioning, multiple-choice questions (MCQs),
and binary classification. Through rigorous benchmarking and ablation
experiments, we reveal key insights for guiding architectural choices when
incorporating multimodal capabilities into LLMs. However, current approaches
have limitations; they do not sufficiently address the need for a diverse
multimodal instruction dataset, which is crucial for enhancing task
generalization. Additionally, they overlook issues related to truthfulness and
factuality when generating responses. These findings illuminate current
methodological constraints in adapting language models for image comprehension
and provide valuable guidance for researchers and practitioners seeking to
harness multimodal versions of LLMs.

该研究分析了不同的多模态指导调优方法，并评估了它们在复杂推理、对话、图像字幕、多项选择题和二元分类等任务中的性能，揭示了在将多模态能力融入大型语言模型时的架构选择的关键见解，但当前方法存在局限性，未能充分解决丰富多样的多模态指导数据集的需求和生成响应的真实性和事实性问题，这些发现阐明了适应图像理解的语言模型的现有方法学限制，并为寻求利用多模态版本的大型语言模型的研究人员和实践者提供了有价值的指导。