This paper proposes a method to gain extra supervision via multi-task
learning for multi-modal video question answering. Multi-modal video question
answering is an important task that aims at the joint understanding of vision
and language. However, establishing large scale dataset for multi-modal video
question answering is expensive and the existing benchmarks are relatively
small to provide sufficient supervision. To overcome this challenge, this paper
proposes a multi-task learning method which is composed of three main
components: (1) multi-modal video question answering network that answers the
question based on the both video and subtitle feature, (2) temporal retrieval
network that predicts the time in the video clip where the question was
generated from and (3) modality alignment network that solves metric learning
problem to find correct association of video and subtitle modalities. By
simultaneously solving related auxiliary tasks with hierarchically shared
intermediate layers, the extra synergistic supervisions are provided. Motivated
by curriculum learning, multi task ratio scheduling is proposed to learn easier
task earlier to set inductive bias at the beginning of the training. The
experiments on publicly available dataset TVQA shows state-of-the-art results,
and ablation studies are conducted to prove the statistical validity.

该研究提出通过多任务学习获得额外监督的方法，解决多模态视频问答建立大规模数据集昂贵且现有基准数据集较小的挑战。该方法由多模态视频问答网络、时间检索网络和模态对齐网络组成，通过层次共享中间层同时解决相关的辅助任务提供额外的协同监督，并提出多任务比率调度的方法进行训练。实验结果表明，该方法在公开数据集 TVQA 上实现了最先进的效果。