Video question answering (VideoQA) is a complex task that requires diverse
multi-modal data for training. Manual annotation of question and answers for
videos, however, is tedious and prohibits scalability. To tackle this problem,
recent methods consider zero-shot settings with no manual annotation of visual
question-answer. In particular, a promising approach adapts frozen
autoregressive language models pretrained on Web-scale text-only data to
multi-modal inputs. In contrast, we here build on frozen bidirectional language
models (BiLM) and show that such an approach provides a stronger and cheaper
alternative for zero-shot VideoQA. In particular, (i) we combine visual inputs
with the frozen BiLM using light trainable modules, (ii) we train such modules
using Web-scraped multi-modal data, and finally (iii) we perform zero-shot
VideoQA inference through masked language modeling, where the masked text is
the answer to a given question. Our proposed approach, FrozenBiLM, outperforms
the state of the art in zero-shot VideoQA by a significant margin on a variety
of datasets, including LSMDC-FiB, iVQA, MSRVTT-QA, MSVD-QA, ActivityNet-QA,
TGIF-FrameQA, How2QA and TVQA. It also demonstrates competitive performance in
the few-shot and fully-supervised setting. Our code and models are publicly
available at this https URL.

本文介绍了一种基于冻结的双向语言模型的零样本视频问答方法，它使用轻量级的可训练模块将可视化输入与冻结的双向语言模型相结合，通过屏蔽的语言模型进行零样本视频问答推理，相较于目前现有的方法，在包括 LSMDC-FiB、iVQA、MSRVTT-QA、MSVD-QA、ActivityNet-QA、TGIF-FrameQA、How2QA 和 TVQA 等各种数据集上，取得了显著的优势，同时在少样本和完全监督的情况下也表现出了有竞争力的结果。