Video QA challenges modelers in multiple fronts. Modeling video necessitates
building not only spatio-temporal models for the dynamic visual channel but
also multimodal structures for associated information channels such as
subtitles or audio. Video QA adds at least two more layers of complexity -
selecting relevant content for each channel in the context of the linguistic
query, and composing spatio-temporal concepts and relations in response to the
query. To address these requirements, we start with two insights: (a) content
selection and relation construction can be jointly encapsulated into a
conditional computational structure, and (b) video-length structures can be
composed hierarchically. For (a) this paper introduces a general-reusable
neural unit dubbed Conditional Relation Network (CRN) taking as input a set of
tensorial objects and translating into a new set of objects that encode
relations of the inputs. The generic design of CRN helps ease the common
complex model building process of Video QA by simple block stacking with
flexibility in accommodating input modalities and conditioning features across
both different domains. As a result, we realize insight (b) by introducing
Hierarchical Conditional Relation Networks (HCRN) for Video QA. The HCRN
primarily aims at exploiting intrinsic properties of the visual content of a
video and its accompanying channels in terms of compositionality, hierarchy,
and near and far-term relation. HCRN is then applied for Video QA in two forms,
short-form where answers are reasoned solely from the visual content, and
long-form where associated information, such as subtitles, presented. Our
rigorous evaluations show consistent improvements over SOTAs on well-studied
benchmarks including large-scale real-world datasets such as TGIF-QA and TVQA,
demonstrating the strong capabilities of our CRN unit and the HCRN for complex
domains such as Video QA.

该论文主要介绍了一种基于条件计算结构的一般性可重用神经元 CRN 和视频 QA 中的分层条件关系网络 HCRN，旨在解决视频问题答案推理的问题。并在广泛的真实世界数据集上展示了其优越性能。

层次化条件关系网络用于多模态视频问答

Hierarchical Conditional Relation Networks for Multimodal Video Question  Answering

Video question answering (VideoQA) is challenging as it requires modeling
capacity to distill dynamic visual artifacts and distant relations and to
associate them with linguistic concepts. We introduce a general-purpose
reusable neural unit called Conditional Relation Network (CRN) that serves as a
building block to construct more sophisticated structures for representation
and reasoning over video. CRN takes as input an array of tensorial objects and
a conditioning feature, and computes an array of encoded output objects. Model
building becomes a simple exercise of replication, rearrangement and stacking
of these reusable units for diverse modalities and contextual information. This
design thus supports high-order relational and multi-step reasoning. The
resulting architecture for VideoQA is a CRN hierarchy whose branches represent
sub-videos or clips, all sharing the same question as the contextual condition.
Our evaluations on well-known datasets achieved new SoTA results, demonstrating
the impact of building a general-purpose reasoning unit on complex domains such
as VideoQA.

通过介绍一种条件关系网络的方法来构建可复用神经元，用于视频中的表示和推理，并在 Well-known 的数据集上取得最新的状态技术，证明 CRN 对于视频问答等复杂域的表示和推理是成功的。