This review paper explores Multimodal Large Language Models (MLLMs), which
integrate Large Language Models (LLMs) like GPT-4 to handle multimodal data
such as text and vision. MLLMs demonstrate capabilities like generating image
narratives and answering image-based questions, bridging the gap towards
real-world human-computer interactions and hinting at a potential pathway to
artificial general intelligence. However, MLLMs still face challenges in
processing the semantic gap in multimodality, which may lead to erroneous
generation, posing potential risks to society. Choosing the appropriate
modality alignment method is crucial, as improper methods might require more
parameters with limited performance improvement. This paper aims to explore
modality alignment methods for LLMs and their existing capabilities.
Implementing modality alignment allows LLMs to address environmental issues and
enhance accessibility. The study surveys existing modal alignment methods in
MLLMs into four groups: (1) Multimodal Converters that change data into
something LLMs can understand; (2) Multimodal Perceivers to improve how LLMs
perceive different types of data; (3) Tools Assistance for changing data into
one common format, usually text; and (4) Data-Driven methods that teach LLMs to
understand specific types of data in a dataset. This field is still in a phase
of exploration and experimentation, and we will organize and update various
existing research methods for multimodal information alignment.

本综述论文探讨了多模态大型语言模型（MLLMs），该模型整合了类似于 GPT-4 的大型语言模型（LLMs），用于处理文本和视觉等多模态数据。MLLMs 展示了生成图像叙述和回答基于图像的问题等能力，缩小了人与计算机之间的差距，并暗示了通向人工智能的潜在途径。然而，MLLMs 仍面临处理多模态语义差距的挑战，可能导致错误生成，对社会造成潜在风险。选择适当的模态对齐方法至关重要，因为不恰当的方法可能需要更多参数，并且性能改进有限。该论文旨在探讨 LLMs 的模态对齐方法及其现有能力。实施模态对齐使得 LLMs 能够解决环境问题并提高可访问性。研究调查了 MLLMs 中现有的模态对齐方法，分为四个组：（1）多模态转换器将数据转换为 LLMs 可以理解的格式；（2）多模态感知器改善 LLMs 对不同类型数据的感知能力；（3）工具辅助将数据转换为一种常见格式，通常是文本；以及（4）数据驱动方法教导 LLMs 理解数据集中特定类型的数据。这个领域仍处于探索和试验阶段，我们将组织和更新各种现有的多模态信息对齐研究方法。