Video Question Answering (Video QA) requires fine-grained understanding of
both video and language modalities to answer the given questions. In this
paper, we propose novel training schemes for multiple-choice video question
answering with a self-supervised pre-training stage and a supervised
contrastive learning in the main stage as an auxiliary learning. In the
self-supervised pre-training stage, we transform the original problem format of
predicting the correct answer into the one that predicts the relevant question
to provide a model with broader contextual inputs without any further dataset
or annotation. For contrastive learning in the main stage, we add a masking
noise to the input corresponding to the ground-truth answer, and consider the
original input of the ground-truth answer as a positive sample, while treating
the rest as negative samples. By mapping the positive sample closer to the
masked input, we show that the model performance is improved. We further employ
locally aligned attention to focus more effectively on the video frames that
are particularly relevant to the given corresponding subtitle sentences. We
evaluate our proposed model on highly competitive benchmark datasets related to
multiple-choice video QA: TVQA, TVQA+, and DramaQA. Experimental results show
that our model achieves state-of-the-art performance on all datasets. We also
validate our approaches through further analyses.

本文介绍了一种新的多项选择视频问题回答（Video QA）的训练方案，其中包括自监督预训练阶段和监督对比学习的辅助学习。在自监督预训练阶段中，我们将原始问题格式进行了转化，以预测相关问题并提供模型更广泛的背景输入。然后，我们在对比学习的主阶段中添加了屏蔽噪声，并通过将正样本映射到受掩输入来改进模型表现。最后，我们采用了本地对齐注意力来更有效地关注相关字幕句子的视频帧。我们的实验结果表明，我们的模型在相关测试集上均取得了最先进的性能，并通过进一步的分析验证了我们的方法。