Despite the impressive capabilities of Multimodal Large Language Models
(MLLMs) in integrating text and image modalities, challenges remain in
accurately interpreting detailed visual elements. This paper presents an
empirical study on enhancing MLLMs with state-of-the-art (SOTA) object
detection and Optical Character Recognition models to improve fine-grained
image understanding and reduce hallucination in responses. Our research
investigates the embedding-based infusion of detection information, the impact
of such infusion on the MLLMs' original abilities, and the interchangeability
of detection models. We conduct systematic experiments with models such as
LLaVA-1.5, DINO, and PaddleOCRv2, revealing that our approach not only refines
MLLMs' performance in specific visual tasks but also maintains their original
strengths. The resulting enhanced MLLMs outperform SOTA models on 9 out of 10
benchmarks, achieving an improvement of up to 12.99% on the normalized average
score, marking a notable advancement in multimodal understanding. We release
our codes to facilitate further exploration into the fine-grained multimodal
dialogue capabilities of MLLMs.

通过融合先进的目标检测和光学字符识别模型，我们在多模态大型语言模型的基础上进行了实证研究，以改善对细粒度图像理解的能力并减少回应中的虚构现象。我们的研究探讨了融合检测信息的基于嵌入的方法，该方法对原始多模态大型语言模型的影响以及检测模型的互换性。通过与 LLaVA-1.5、DINO 和 PaddleOCRv2 等模型进行系统实验，我们发现我们的方法不仅在特定的视觉任务中提高了多模态大型语言模型的性能，而且保持了其原始优势。结果表明，改进后的多模态大型语言模型在 10 个基准测试中有 9 个超过了先进模型，在规范化的平均得分上取得了最高 12.99% 的提升，标志着多模态理解领域的重大进展。通过发布我们的代码，我们希望进一步探索多模态大型语言模型在细粒度多模态对话能力方面的应用。