Chain-of-thought (CoT) reasoning has exhibited impressive performance in
language models for solving complex tasks and answering questions. However,
many real-world questions require multi-modal information, such as text and
images. Previous research on multi-modal CoT has primarily focused on
extracting fixed image features from off-the-shelf vision models and then
fusing them with text using attention mechanisms. This approach has limitations
because these vision models were not designed for complex reasoning tasks and
do not align well with language thoughts. To overcome this limitation, we
introduce a novel approach for multi-modal CoT reasoning that utilizes latent
space learning via diffusion processes to generate effective image features
that align with language thoughts. Our method fuses image features and text
representations at a deep level and improves the complex reasoning ability of
multi-modal CoT. We demonstrate the efficacy of our proposed method on
multi-modal ScienceQA and machine translation benchmarks, achieving
state-of-the-art performance on ScienceQA. Overall, our approach offers a more
robust and effective solution for multi-modal reasoning in language models,
enhancing their ability to tackle complex real-world problems.

我们提出了一种新的多模态链式思考推理方法，通过扩散过程利用潜在空间学习产生与语言思维相吻合的有效图像特征来融合图像特征和文本表示，提高多模态链式思考推理的复杂推理能力，从而为语言模型在多模态推理中提供了更健壮和有效的解决方案。

多模态潜空间学习用于语言模型的思维链推理

Multi-modal Latent Space Learning for Chain-of-Thought Reasoning in  Language Models

It is practical to assume that an individual view is unlikely to be
sufficient for effective multi-view learning. Therefore, integration of
multi-view information is both valuable and necessary. In this paper, we
propose the Multi-view Intact Space Learning (MISL) algorithm, which integrates
the encoded complementary information in multiple views to discover a latent
intact representation of the data. Even though each view on its own is
insufficient, we show theoretically that by combing multiple views we can
obtain abundant information for latent intact space learning. Employing the
Cauchy loss (a technique used in statistical learning) as the error measurement
strengthens robustness to outliers. We propose a new definition of multi-view
stability and then derive the generalization error bound based on multi-view
stability and Rademacher complexity, and show that the complementarity between
multiple views is beneficial for the stability and generalization. MISL is
efficiently optimized using a novel Iteratively Reweight Residuals (IRR)
technique, whose convergence is theoretically analyzed. Experiments on
synthetic data and real-world datasets demonstrate that MISL is an effective
and promising algorithm for practical applications.

本文提出了一种基于多视角信息融合的多视角完整空间学习（MISL）算法，利用编码后的互补信息来发现数据的潜在完整性表示。实验证明，MISL 是一种有效的实用算法，其稳定性和泛化能力都得到了保证。