Multimodal Large Language Models (MLLMs) are experiencing rapid growth,
yielding a plethora of noteworthy contributions in recent months. The
prevailing trend involves adopting data-driven methodologies, wherein diverse
instruction-following datasets are collected. However, a prevailing challenge
persists in these approaches, specifically in relation to the limited visual
perception ability, as CLIP-like encoders employed for extracting visual
information from inputs. Though these encoders are pre-trained on billions of
image-text pairs, they still grapple with the information loss dilemma, given
that textual captions only partially capture the contents depicted in images.
To address this limitation, this paper proposes to improve the visual
perception ability of MLLMs through a mixture-of-experts knowledge enhancement
mechanism. Specifically, we introduce a novel method that incorporates
multi-task encoders and visual tools into the existing MLLMs training and
inference pipeline, aiming to provide a more comprehensive and accurate
summarization of visual inputs. Extensive experiments have evaluated its
effectiveness of advancing MLLMs, showcasing improved visual perception
achieved through the integration of visual experts.

这篇论文提出了一种通过专家混合知识增强机制来改善多模态大型语言模型 (MLLMs) 的视觉感知能力的方法，并通过集成视觉专家实现了视觉输入的更全面准确的概括，进一步提升了 MLLMs 的视觉感知能力。