Despite significant progress in video question answering (VideoQA), existing
methods fall short of questions that require causal/temporal reasoning across
frames. This can be attributed to imprecise motion representations. We
introduce Action Temporality Modeling (ATM) for temporality reasoning via
three-fold uniqueness: (1) rethinking the optical flow and realizing that
optical flow is effective in capturing the long horizon temporality reasoning;
(2) training the visual-text embedding by contrastive learning in an
action-centric manner, leading to better action representations in both vision
and text modalities; and (3) preventing the model from answering the question
given the shuffled video in the fine-tuning stage, to avoid spurious
correlation between appearance and motion and hence ensure faithful temporality
reasoning. In the experiments, we show that ATM outperforms previous approaches
in terms of the accuracy on multiple VideoQAs and exhibits better true
temporality reasoning ability.

通过动作时间建模（ATM），在视频问答（VideoQA）中引入了对因果 / 时间推理跨帧的问题，并通过重新思考光流的有效性、以行为为中心的对比学习和防止模型在微调阶段给出对洗牌视频的回答来实现了时序推理。实验证明，ATM 在多个 VideoQA 任务的准确性方面优于先前方法，并展示了更好的真实时间推理能力。