The increasing demand for intelligent systems capable of interpreting and
reasoning about visual content requires the development of Large Multi-Modal
Models (LMMs) that are not only accurate but also have explicit reasoning
capabilities. This paper presents a novel approach to imbue an LMM with the
ability to conduct explicit reasoning based on visual content and textual
instructions. We introduce a system that can ask a question to acquire
necessary knowledge, thereby enhancing the robustness and explicability of the
reasoning process. Our method comprises the development of a novel dataset
generated by a Large Language Model (LLM), designed to promote chain-of-thought
reasoning combined with a question-asking mechanism. We designed an LMM, which
has high capabilities on region awareness to address the intricate requirements
of image-text alignment. The model undergoes a three-stage training phase,
starting with large-scale image-text alignment using a large-scale datasets,
followed by instruction tuning, and fine-tuning with a focus on
chain-of-thought reasoning. The results demonstrate a stride toward a more
robust, accurate, and interpretable LMM, capable of reasoning explicitly and
seeking information proactively when confronted with ambiguous visual input.

本文介绍了一种通过图像内容和文字指令进行显式推理的新方法，采用大型多模态模型（LMM），并引入了一个能够提问以获取必要知识以增强推理过程鲁棒性和可解释性的系统。实验结果表明我们的方法能够在面对模糊的视觉输入时，朝着更具鲁棒性、准确性和可解释性的 LMM 迈进。