While large multi-modal models (LMMs) have exhibited impressive capabilities
across diverse tasks, their effectiveness in handling complex tasks has been
limited by the prevailing single-step reasoning paradigm. To this end, this
paper proposes VoCoT, a multi-step Visually grounded object-centric
Chain-of-Thought reasoning framework tailored for inference with LMMs. VoCoT is
characterized by two key features: (1) object-centric reasoning paths that
revolve around cross-modal shared object-level information, and (2) visually
grounded representation of object concepts in a multi-modal interleaved and
aligned manner, which effectively bridges the modality gap within LMMs during
long-term generation. Additionally, we construct an instruction dataset to
facilitate LMMs in adapting to reasoning with VoCoT. By introducing VoCoT into
the prevalent open-source LMM architecture, we introduce VolCano. With only 7B
parameters and limited input resolution, VolCano demonstrates excellent
performance across various scenarios, surpassing SOTA models, including GPT-4V,
in tasks requiring complex reasoning. Our code, data and model will be
available at this https URL

该研究论文提出了 VoCoT，一种适合于大型多模态模型推理的多步骤可视化物体为中心的连贯思维推理框架，通过在 LMMs 中引入 VoCoT，实现了在长期生成过程中跨模态信息的有效连接，并在各种场景中表现出优秀的性能，超过了需要复杂推理的 SOTA 模型，包括 GPT-4V。

VoCoT: 大型多模态模型中释放基于视觉基础的多步推理

VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large  Multi-Modal Models

Large multimodal models (LMMs) suffer from multimodal hallucination, where
they provide incorrect responses misaligned with the given visual information.
Recent works have conjectured that one of the reasons behind multimodal
hallucination might be due to the vision encoder failing to ground on the image
properly. To mitigate this issue, we propose a novel approach that leverages
self-feedback as visual cues. Building on this approach, we introduce Volcano,
a multimodal self-feedback guided revision model. Volcano generates natural
language feedback to its initial response based on the provided visual
information and utilizes this feedback to self-revise its initial response.
Volcano effectively reduces multimodal hallucination and achieves
state-of-the-art on MMHal-Bench, POPE, and GAVIE. It also improves on general
multimodal abilities and outperforms previous models on MM-Vet and MMBench.
Through a qualitative analysis, we show that Volcano's feedback is properly
grounded on the image than the initial response. This indicates that Volcano
can provide itself with richer visual information, helping alleviate multimodal
hallucination. We publicly release Volcano models of 7B and 13B sizes along
with the data and code at this https URL

通过自我反馈引导修订模型，Volcano 可以有效降低多模态幻觉，并在多项基准测试中达到最新水平，通过定性分析，我们展示了 Volcano 反馈信息与初始响应相比，更好地与图像对应。