Large Language Models (LLMs) have shown remarkable performances on a wide
range of natural language understanding and generation tasks. We observe that
the LLMs provide effective priors in exploiting $\textit{linguistic shortcuts}$
for temporal and causal reasoning in Video Question Answering (VideoQA).
However, such priors often cause suboptimal results on VideoQA by leading the
model to over-rely on questions, $\textit{i.e.}$, $\textit{linguistic bias}$,
while ignoring visual content. This is also known as `ungrounded guesses' or
`hallucinations'. To address this problem while leveraging LLMs' prior on
VideoQA, we propose a novel framework, Flipped-VQA, encouraging the model to
predict all the combinations of $\langle$V, Q, A$\rangle$ triplet by flipping
the source pair and the target label to understand their complex relationships,
$\textit{i.e.}$, predict A, Q, and V given a VQ, VA, and QA pairs,
respectively. In this paper, we develop LLaMA-VQA by applying Flipped-VQA to
LLaMA, and it outperforms both LLMs-based and non-LLMs-based models on five
challenging VideoQA benchmarks. Furthermore, our Flipped-VQA is a general
framework that is applicable to various LLMs (OPT and GPT-J) and consistently
improves their performances. We empirically demonstrate that Flipped-VQA not
only enhances the exploitation of linguistic shortcuts but also mitigates the
linguistic bias, which causes incorrect answers over-relying on the question.
Code is available at this https URL

通过使用 Flipped-VQA 框架，我们成功应用于 LLaMA-VQA 和其它 LLMs 模型，取得了在五个具有挑战性的 VideoQA 基准测试中优于基于 LLMs 和非 LLMs 模型的结果，并且实验证明 Flipped-VQA 不仅增强了语言快捷方式的利用，还减轻了由于过度依赖问题而导致错误答案的语言偏见。