Video question answering (VideoQA) is challenging given its multimodal
combination of visual understanding and natural language understanding. While
existing approaches seldom leverage the appearance-motion information in the
video at multiple temporal scales, the interaction between the question and the
visual information for textual semantics extraction is frequently ignored.
Targeting these issues, this paper proposes a novel Temporal Pyramid
Transformer (TPT) model with multimodal interaction for VideoQA. The TPT model
comprises two modules, namely Question-specific Transformer (QT) and Visual
Inference (VI). Given the temporal pyramid constructed from a video, QT builds
the question semantics from the coarse-to-fine multimodal co-occurrence between
each word and the visual content. Under the guidance of such question-specific
semantics, VI infers the visual clues from the local-to-global multi-level
interactions between the question and the video. Within each module, we
introduce a multimodal attention mechanism to aid the extraction of
question-video interactions, with residual connections adopted for the
information passing across different levels. Through extensive experiments on
three VideoQA datasets, we demonstrate better performances of the proposed
method in comparison with the state-of-the-arts.

本文提出了一种基于时域金字塔变换器的多模交互模型，用于视频问答，实现了问题语义的建立和视频信息的推断，并在三个视频问答数据集上通过了广泛的实验证明了该方法相较于现有技术的卓越表现。