While VideoQA Transformer models demonstrate competitive performance on
standard benchmarks, the reasons behind their success remain unclear. Do these
models jointly capture and leverage the rich multimodal structures and dynamics
from video and text? Or are they merely exploiting shortcuts to achieve high
scores? We analyze this with $\textit{QUAG}$ (QUadrant AveraGe), a lightweight
and non-parametric probe that systematically ablates the model's coupled
multimodal understanding during inference. Surprisingly, QUAG reveals that the
models manage to maintain high performance even when injected with multimodal
sub-optimality. Additionally, even after replacing self-attention in multimodal
fusion blocks with "QUAG-attention", a simplistic and less-expressive variant
of self-attention, the models maintain high performance. This means that
current VideoQA benchmarks and their metrics do not penalize shortcuts that
discount joint multimodal understanding. Motivated by this, we propose the
$\textit{CLAVI}$ (Counterfactual in LAnguage and VIdeo) benchmark, a diagnostic
dataset for benchmarking coupled multimodal understanding in VideoQA through
counterfactuals. CLAVI consists of temporal questions and videos that are
augmented to curate balanced counterfactuals in language and video domains.
Hence, it incentivizes, and identifies the reliability of learnt multimodal
representations. We evaluate CLAVI and find that models achieve high
performance on multimodal shortcut instances, but have very poor performance on
the counterfactuals. Hence, we position CLAVI as a litmus test to identify,
diagnose and improve the sub-optimality of learnt multimodal VideoQA
representations which the current benchmarks are unable to assess.

通过 QUAG 探究 VideoQA Transformer 模型对于多模态结构与动态的理解，提出对于多模态 VideoQA 表示的可靠性检验，CLAVI Benchmark, 通过对反事实情景的测评，发现目前的模型在处理多模态 VideoQA 表示的亚优化方面有待改善。