Video question answering is a challenging task, which requires agents to be
able to understand rich video contents and perform spatial-temporal reasoning.
However, existing graph-based methods fail to perform multi-step reasoning
well, neglecting two properties of VideoQA: (1) Even for the same video,
different questions may require different amount of video clips or objects to
infer the answer with relational reasoning; (2) During reasoning, appearance
and motion features have complicated interdependence which are correlated and
complementary to each other. Based on these observations, we propose a
Dual-Visual Graph Reasoning Unit (DualVGR) which reasons over videos in an
end-to-end fashion. The first contribution of our DualVGR is the design of an
explainable Query Punishment Module, which can filter out irrelevant visual
features through multiple cycles of reasoning. The second contribution is the
proposed Video-based Multi-view Graph Attention Network, which captures the
relations between appearance and motion features. Our DualVGR network achieves
state-of-the-art performance on the benchmark MSVD-QA and SVQA datasets, and
demonstrates competitive results on benchmark MSRVTT-QA datasets. Our code is
available at this https URL

提出了一个 Dual-Visual Graph Reasoning Unit (DualVGR) 网络，在视频问答中实现可解释的查询惩罚模块同时捕捉外观和运动特征之间的关系，在 MSVD-QA 基准数据集上取得了最佳性能.