This paper considers a network referred to as Modality Shifting Attention
Network (MSAN) for Multimodal Video Question Answering (MVQA) task. MSAN
decomposes the task into two sub-tasks: (1) localization of temporal moment
relevant to the question, and (2) accurate prediction of the answer based on
the localized moment. The modality required for temporal localization may be
different from that for answer prediction, and this ability to shift modality
is essential for performing the task. To this end, MSAN is based on (1) the
moment proposal network (MPN) that attempts to locate the most appropriate
temporal moment from each of the modalities, and also on (2) the heterogeneous
reasoning network (HRN) that predicts the answer using an attention mechanism
on both modalities. MSAN is able to place importance weight on the two
modalities for each sub-task using a component referred to as Modality
Importance Modulation (MIM). Experimental results show that MSAN outperforms
previous state-of-the-art by achieving 71.13\% test accuracy on TVQA benchmark
dataset. Extensive ablation studies and qualitative analysis are conducted to
validate various components of the network.

该论文提出了一种称为模态转移关注网络的模型解决多模式视频问答任务，该模型对任务进行任务分解并使用时间提议网络和异构推理网络进行处理，通过模态转移进行任务的执行，这种模型在 TVQA 基准数据集上的实验结果表明它优于之前的最优性能，达到 71.13％的测试准确率，并开展了广泛的消融研究和定性分析以验证网络的各个组件。