This paper considers a network referred to as Modality Shifting Attention
Network (MSAN) for Multimodal Video Question Answering (MVQA) task. MSAN
decomposes the task into two sub-tasks: (1) localization of temporal moment
relevant to the question, and (2) accurate prediction of the answer based on
the localized moment. The modality required for temporal localization may be
different from that for answer prediction, and this ability to shift modality
is essential for performing the task. To this end, MSAN is based on (1) the
moment proposal network (MPN) that attempts to locate the most appropriate
temporal moment from each of the modalities, and also on (2) the heterogeneous
reasoning network (HRN) that predicts the answer using an attention mechanism
on both modalities. MSAN is able to place importance weight on the two
modalities for each sub-task using a component referred to as Modality
Importance Modulation (MIM). Experimental results show that MSAN outperforms
previous state-of-the-art by achieving 71.13\% test accuracy on TVQA benchmark
dataset. Extensive ablation studies and qualitative analysis are conducted to
validate various components of the network.

该论文提出了一种称为模态转移关注网络的模型解决多模式视频问答任务，该模型对任务进行任务分解并使用时间提议网络和异构推理网络进行处理，通过模态转移进行任务的执行，这种模型在 TVQA 基准数据集上的实验结果表明它优于之前的最优性能，达到 71.13％的测试准确率，并开展了广泛的消融研究和定性分析以验证网络的各个组件。

多模态视频问答的情感转移注意力网络

Modality Shifting Attention Network for Multi-modal Video Question  Answering

This paper studies the problem of temporal moment localization in a long
untrimmed video using natural language as the query. Given an untrimmed video
and a sentence as the query, the goal is to determine the starting, and the
ending, of the relevant visual moment in the video, that corresponds to the
query sentence. While previous works have tackled this task by a
propose-and-rank approach, we introduce a more efficient, end-to-end trainable,
and {\em proposal-free approach} that relies on three key components: a dynamic
filter to transfer language information to the visual domain, a new loss
function to guide our model to attend the most relevant parts of the video, and
soft labels to model annotation uncertainty. We evaluate our method on two
benchmark datasets, Charades-STA and ActivityNet-Captions. Experimental results
show that our approach outperforms state-of-the-art methods on both datasets.

本文提出了一种更有效率的、端到端可训练的、不需要先提出建议的方法，以自然语言作为查询，解决了在长视频中定位时间点的问题。通过引入动态滤波器、新的损失函数和软标签等三个关键组件，实现从语言信息到视觉领域的转换，并评估了该方法在两个基准数据集上的表现优越性。