The remarkable performance of Multimodal Large Language Models (MLLMs) has
unequivocally demonstrated their proficient understanding capabilities in
handling a wide array of visual tasks. Nevertheless, the opaque nature of their
black-box reasoning processes persists as an enigma, rendering them
uninterpretable and struggling with hallucination. Their ability to execute
intricate compositional reasoning tasks is also constrained, culminating in a
stagnation of learning progression for these models. In this work, we introduce
Fact, a novel paradigm designed to generate multimodal rationales that are
faithful, concise, and transferable for teaching MLLMs. This paradigm utilizes
verifiable visual programming to generate executable code guaranteeing
faithfulness and precision. Subsequently, through a series of operations
including pruning, merging, and bridging, the rationale enhances its
conciseness. Furthermore, we filter rationales that can be transferred to
end-to-end paradigms from programming paradigms to guarantee transferability.
Empirical evidence from experiments demonstrates the superiority of our method
across models of varying parameter sizes, significantly enhancing their
compositional reasoning and generalization ability. Our approach also reduces
hallucinations owing to its high correlation between images and text.

在这项研究中，我们引入了一种名为 Fact 的新范式，旨在生成忠实、简洁和可传递的多模态理由，通过可验证的视觉编程生成可执行的代码，从而增强了它的忠实性和精确性，并通过剪枝、合并和桥接等一系列操作提高了简洁性，同时通过过滤可以从编程范式传递到端到端范式的理由来保证可传递性，实验证据表明我们的方法在各种参数大小的模型上都表现出优异的组合推理和泛化能力，而且由于图像和文本之间的高相关性，我们的方法还减少了错觉。