The goal of visual answering localization (VAL) in the video is to obtain a
relevant and concise time clip from a video as the answer to the given natural
language question. Early methods are based on the interaction modelling between
video and text to predict the visual answer by the visual predictor. Later,
using the textual predictor with subtitles for the VAL proves to be more
precise. However, these existing methods still have cross-modal knowledge
deviations from visual frames or textual subtitles. In this paper, we propose a
cross-modal mutual knowledge transfer span localization (MutualSL) method to
reduce the knowledge deviation. MutualSL has both visual predictor and textual
predictor, where we expect the prediction results of these both to be
consistent, so as to promote semantic knowledge understanding between
cross-modalities. On this basis, we design a one-way dynamic loss function to
dynamically adjust the proportion of knowledge transfer. We have conducted
extensive experiments on three public datasets for evaluation. The experimental
results show that our method outperforms other competitive state-of-the-art
(SOTA) methods, demonstrating its effectiveness.

本文提出了一种跨模态互相知识传递的跨模态互知跨度本地化方法 (MutualSL)，它包含视觉预测器和文本预测器两个部分，旨在通过构建一种双向动态损失函数以调整传递比例来提高语义知识的理解，以获取视频中自然语言问题的相关和简明时间剪辑答案。实验结果表明，该方法优于其他竞争同类方法，展示了其有效性。

跨模态相互知识迁移的视觉答案定位

Visual Answer Localization with Cross-modal Mutual Knowledge Transfer

We propose a scalable approach to learn video-based question answering (QA):
answer a "free-form natural language question" about a video content. Our
approach automatically harvests a large number of videos and descriptions
freely available online. Then, a large number of candidate QA pairs are
automatically generated from descriptions rather than manually annotated. Next,
we use these candidate QA pairs to train a number of video-based QA methods
extended fromMN (Sukhbaatar et al. 2015), VQA (Antol et al. 2015), SA (Yao et
al. 2015), SS (Venugopalan et al. 2015). In order to handle non-perfect
candidate QA pairs, we propose a self-paced learning procedure to iteratively
identify them and mitigate their effects in training. Finally, we evaluate
performance on manually generated video-based QA pairs. The results show that
our self-paced learning procedure is effective, and the extended SS model
outperforms various baselines.

本文提出了一种可扩展的视频问答技术，利用自动生成的大量候选问题 - 答案对并使用自适应学习方法以处理其中存在的非理想数据，取得了优于基线模型的效果。

利用视频描述学习视频问答

Leveraging Video Descriptions to Learn Video Question Answering

Suppose we want to build a system that answers a natural language question by
representing its semantics as a logical form and computing the answer given a
structured database of facts. The core part of such a system is the semantic
parser that maps questions to logical forms. Semantic parsers are typically
trained from examples of questions annotated with their target logical forms,
but this type of annotation is expensive.
Our goal is to learn a semantic parser from question-answer pairs instead,
where the logical form is modeled as a latent variable. Motivated by this
challenging learning problem, we develop a new semantic formalism,
dependency-based compositional semantics (DCS), which has favorable linguistic,
statistical, and computational properties. We define a log-linear distribution
over DCS logical forms and estimate the parameters using a simple procedure
that alternates between beam search and numerical optimization. On two standard
semantic parsing benchmarks, our system outperforms all existing
state-of-the-art systems, despite using no annotated logical forms.

研究使用问题 - 回答对来学习语义解析器，其中逻辑形式被建模为潜变量，并提出了一种新的语义形式 —— 基于依赖的组成语义学。该方法具有有利的语言学、统计和计算性质，并在两个标准语义解析基准测试中优于所有现有的最先进系统。