While Multi-modal Language Models (MLMs) demonstrate impressive multimodal
ability, they still struggle on providing factual and precise responses for
tasks like visual question answering (VQA). In this paper, we address this
challenge from the perspective of contextual information. We propose Causal
Context Generation, Causal-CoG, which is a prompting strategy that engages
contextual information to enhance precise VQA during inference. Specifically,
we prompt MLMs to generate contexts, i.e, text description of an image, and
engage the generated contexts for question answering. Moreover, we investigate
the advantage of contexts on VQA from a causality perspective, introducing
causality filtering to select samples for which contextual information is
helpful. To show the effectiveness of Causal-CoG, we run extensive experiments
on 10 multimodal benchmarks and show consistent improvements, e.g., +6.30% on
POPE, +13.69% on Vizwiz and +6.43% on VQAv2 compared to direct decoding,
surpassing existing methods. We hope Casual-CoG inspires explorations of
context knowledge in multimodal models, and serves as a plug-and-play strategy
for MLM decoding.

我们提出了一种从上下文信息的角度解决多模态语言模型在视觉问答任务中难以提供准确和精确回答的挑战的方法：因果上下文生成策略（Causal-CoG）。我们通过提示多模态语言模型生成上下文，即图像的文本描述，并利用生成的上下文进行问题回答，同时从因果关系的角度研究上下文在视觉问答中的优势，引入因果过滤来选择上下文信息对于有帮助的样本。通过在 10 个多模态基准测试上进行广泛实验，与直接解码相比，Causal-CoG 的有效性得到证明，例如在 POPE 测试上提高了 6.30%，在 Vizwiz 测试上提高了 13.69%，在 VQAv2 测试上提高了 6.43%，超过了现有方法。我们希望 Causal-CoG 能够激发对多模态模型中上下文知识的探索，并作为多模态语言模型解码的即插即用策略。