This paper presents a new method for end-to-end Video Question Answering
(VideoQA), aside from the current popularity of using large-scale pre-training
with huge feature extractors. We achieve this with a pyramidal multimodal
transformer (PMT) model, which simply incorporates a learnable word embedding
layer, a few convolutional and transformer layers. We use the anisotropic
pyramid to fulfill video-language interactions across different spatio-temporal
scales. In addition to the canonical pyramid, which includes both bottom-up and
top-down pathways with lateral connections, novel strategies are proposed to
decompose the visual feature stream into spatial and temporal sub-streams at
different scales and implement their interactions with the linguistic semantics
while preserving the integrity of local and global semantics. We demonstrate
better or on-par performances with high computational efficiency against
state-of-the-art methods on five VideoQA benchmarks. Our ablation study shows
the scalability of our model that achieves competitive results for
text-to-video retrieval by leveraging feature extractors with reusable
pre-trained weights, and also the effectiveness of the pyramid.

本文提出了一种新的端到端视频问答方法，使用金字塔多模态变换器（PMT）模型实现视频语言交互，通过使用异性金字塔在不同的时空尺度上实现跨视频语言交互，并在保持本地和全局语义完整性的同时，将视频特征流分解为空间和时间子流，并实现它们与语义的交互，并结合可重用的预训练权重的特征提取器，在 5 个视频问答基准测试中实现更好或同等的性能。