Recent advances in large language models elicit reasoning in a chain of
thought that allows models to decompose problems in a human-like fashion.
Though this paradigm improves multi-step reasoning ability in language models,
it is limited by being unimodal and applied mainly to question-answering tasks.
We claim that incorporating visual augmentation into reasoning is essential,
especially for complex, imaginative tasks. Consequently, we introduce VCoT, a
novel method that leverages chain of thought prompting with vision-language
grounding to recursively bridge the logical gaps within sequential data. Our
method uses visual guidance to generate synthetic multimodal infillings that
add consistent and novel information to reduce the logical gaps for downstream
tasks that can benefit from temporal reasoning, as well as provide
interpretability into models' multi-step reasoning. We apply VCoT to the Visual
Storytelling and WikiHow summarization datasets and demonstrate through human
evaluation that VCoT offers novel and consistent synthetic data augmentation
beating chain of thought baselines, which can be used to enhance downstream
performance.

通过视觉增强实现 VCoT 方法，利用多模态填充降低序列数据中的逻辑间隙，改善下游任务的表现及对模型的多步推理提供可解释性。在视觉叙事和 WikiHow 摘要数据集上，VCoT 方法通过人类评估超越了思维链基线模型，提供了新的、一致的合成数据增强。