Multi-modal large language models (MLLMs) are trained based on large language
models (LLM), with an enhanced capability to comprehend multi-modal inputs and
generate textual responses. While they excel in multi-modal tasks, the pure NLP
abilities of MLLMs are often underestimated and left untested. In this study,
we get out of the box and unveil an intriguing characteristic of MLLMs -- our
preliminary results suggest that visual instruction tuning, a prevailing
strategy for transitioning LLMs into MLLMs, unexpectedly and interestingly
helps models attain both improved truthfulness and ethical alignment in the
pure NLP context. For example, a visual-instruction-tuned LLaMA2 7B model
surpasses the performance of the LLaMA2-chat 7B model, fine-tuned with over one
million human annotations, on TruthfulQA-mc and Ethics benchmarks. Further
analysis reveals that the improved alignment can be attributed to the superior
instruction quality inherent to visual-text data. In releasing our code at
github.com/UCSC-VLAA/Sight-Beyond-Text, we aspire to foster further exploration
into the intrinsic value of visual-text synergies and, in a broader scope,
multi-modal interactions in alignment research.

多模态大型语言模型（MLLMs）训练基于大型语言模型（LLM），具备理解多模态输入和生成文本响应的增强能力。本研究表明，视觉指导调整在纯自然语言处理（NLP）环境中意外且有趣地帮助模型实现了改进的真实性和道德一致性。