Chain of thought (CoT) has proven useful for problems requiring complex
reasoning. Many of these problems are both textual and multimodal. Given the
inputs in different modalities, a model generates a rationale and then uses it
to answer a question. Because of the hallucination issue, the generated soft
negative rationales with high textual quality but illogical semantics do not
always help improve answer accuracy. This study proposes a rationale generation
method using soft negative sampling (SNSE-CoT) to mitigate hallucinations in
multimodal CoT. Five methods were applied to generate soft negative samples
that shared highly similar text but had different semantics from the original.
Bidirectional margin loss (BML) was applied to introduce them into the
traditional contrastive learning framework that involves only positive and
negative samples. Extensive experiments on the ScienceQA dataset demonstrated
the effectiveness of the proposed method. Code and data are released at
this https URL

利用软负采样的方法减轻多模式 CoT 中幻觉问题的研究提出了一种理性生成方法（SNSE-CoT）。在传统的对比学习框架中引入了五种方法来生成高度相似但具有不同语义的软负样本，并通过双向边界损失将其应用于多模式 CoT 中，对 ScienceQA 数据集进行了广泛实验，并证明了该方法的有效性。

通过软负采样增强多模态思维链中的语义

Enhancing Semantics in Multimodal Chain of Thought via Soft Negative  Sampling

With the advent of large language models(LLMs) enhanced by the
chain-of-thought(CoT) methodology, visual reasoning problem is usually
decomposed into manageable sub-tasks and tackled sequentially with various
external tools. However, such a paradigm faces the challenge of the potential
"determining hallucinations" in decision-making due to insufficient visual
information and the limitation of low-level perception tools that fail to
provide abstract summaries necessary for comprehensive reasoning. We argue that
converging visual context acquisition and logical reasoning is pivotal for
tackling visual reasoning tasks. This paper delves into the realm of multimodal
CoT to solve intricate visual reasoning tasks with multimodal large language
models(MLLMs) and their cognitive capability. To this end, we propose an
innovative multimodal CoT framework, termed Cantor, characterized by a
perception-decision architecture. Cantor first acts as a decision generator and
integrates visual inputs to analyze the image and problem, ensuring a closer
alignment with the actual context. Furthermore, Cantor leverages the advanced
cognitive functions of MLLMs to perform as multifaceted experts for deriving
higher-level information, enhancing the CoT generation process. Our extensive
experiments demonstrate the efficacy of the proposed framework, showing
significant improvements in multimodal CoT performance across two complex
visual reasoning datasets, without necessitating fine-tuning or ground-truth
rationales. Project Page: this https URL .

通过结合视觉环境获取和逻辑推理，本论文提出了一种名为 Cantor 的创新的多模态 CoT 框架，利用多模态大语言模型的认知能力解决复杂的视觉推理任务，该框架通过分析图像和问题实现对视觉输入的整合，利用大语言模型的高级认知功能生成更高级别的信息，提高了多模态 CoT 性能。