In recent years, multimodal large language models (MLLMs) have shown
remarkable capabilities in tasks like visual question answering and common
sense reasoning, while visual perception models have made significant strides
in perception tasks, such as detection and segmentation. However, MLLMs mainly
focus on high-level image-text interpretations and struggle with fine-grained
visual understanding, and vision perception models usually suffer from
open-world distribution shifts due to their limited model capacity. To overcome
these challenges, we propose the Mutually Reinforced Multimodal Large Language
Model (MR-MLLM), a novel framework that synergistically enhances visual
perception and multimodal comprehension. First, a shared query fusion mechanism
is proposed to harmonize detailed visual inputs from vision models with the
linguistic depth of language models, enhancing multimodal comprehension and
vision perception synergistically. Second, we propose the perception-enhanced
cross-modal integration method, incorporating novel modalities from vision
perception outputs, like object detection bounding boxes, to capture subtle
visual elements, thus enriching the understanding of both visual and textual
data. In addition, an innovative perception-embedded prompt generation
mechanism is proposed to embed perceptual information into the language model's
prompts, aligning the responses contextually and perceptually for a more
accurate multimodal interpretation. Extensive experiments demonstrate MR-MLLM's
superior performance in various multimodal comprehension and vision perception
tasks, particularly those requiring corner case vision perception and
fine-grained language comprehension.

一个名为 Mutually Reinforced Multimodal Large Language Model (MR-MLLM) 的新框架，通过共享查询融合机制和增强的跨模态集成方法，结合视觉感知和多模态理解，以及混合了感知信息的提示生成机制，提供更准确的多模态解释，在各种多模态理解和视觉感知任务中展现卓越性能。