This paper proposes a Video Graph Transformer (VGT) model for Video Quetion
Answering (VideoQA). VGT's uniqueness are two-fold: 1) it designs a dynamic
graph transformer module which encodes video by explicitly capturing the visual
objects, their relations, and dynamics for complex spatio-temporal reasoning;
and 2) it exploits disentangled video and text Transformers for relevance
comparison between the video and text to perform QA, instead of entangled
cross-modal Transformer for answer classification. Vision-text communication is
done by additional cross-modal interaction modules. With more reasonable video
encoding and QA solution, we show that VGT can achieve much better performances
on VideoQA tasks that challenge dynamic relation reasoning than prior arts in
the pretraining-free scenario. Its performances even surpass those models that
are pretrained with millions of external data. We further show that VGT can
also benefit a lot from self-supervised cross-modal pretraining, yet with
orders of magnitude smaller data. These results clearly demonstrate the
effectiveness and superiority of VGT, and reveal its potential for more
data-efficient pretraining. With comprehensive analyses and some heuristic
observations, we hope that VGT can promote VQA research beyond coarse
recognition/description towards fine-grained relation reasoning in realistic
videos. Our code is available at this https URL

该研究论文提出了一种名为 VGT 的视频图形转换器模型，旨在解决视频问答任务中动态关系推理方面的挑战，其独特性在于利用动态图形变换器模块对视频进行编码，并利用分离的视频和文本变换器来执行问答任务。通过充分的分析和启发性的观察，证明了 VGT 的有效性和优越性，并揭示了其可用于更数据有效的预训练。