In multimodal machine learning tasks, it is due to the complexity of the
assignments that the network structure, in most cases, is assembled in a
sophisticated way. The holistic architecture can be separated into several
logical parts according to the respective ends that the modules are devised to
achieve. As the number of modalities of information representation increases,
constructing ad hoc subnetworks for processing the data from divergent
modalities while mediating the fusion of different information types has become
a cumbersome and expensive problem. In this paper, we present an approach to
facilitate the question with a reusable and composable neural unit; by
connecting the units in series or parallel, the arduous network constructing of
multimodal machine learning tasks will be accomplished in a much
straightforward way. Additionally, through parameter sharing (weights
replication) among the units, the space complexity will be significantly
reduced. We have conducted experiments on three commonly used datasets; our
method achieves impressive performance compared to several video QA baselines.

多模态机器学习中的网络结构复杂，通过连接可重用且可组合的神经单元，可以以更直观的方式完成多模态机器学习任务的网络构建，同时通过参数共享大大减少了空间复杂度。

模块化混合注意力网络用于视频问答

Modular Blended Attention Network for Video Question Answering

Video question answering (VideoQA) is challenging as it requires modeling
capacity to distill dynamic visual artifacts and distant relations and to
associate them with linguistic concepts. We introduce a general-purpose
reusable neural unit called Conditional Relation Network (CRN) that serves as a
building block to construct more sophisticated structures for representation
and reasoning over video. CRN takes as input an array of tensorial objects and
a conditioning feature, and computes an array of encoded output objects. Model
building becomes a simple exercise of replication, rearrangement and stacking
of these reusable units for diverse modalities and contextual information. This
design thus supports high-order relational and multi-step reasoning. The
resulting architecture for VideoQA is a CRN hierarchy whose branches represent
sub-videos or clips, all sharing the same question as the contextual condition.
Our evaluations on well-known datasets achieved new SoTA results, demonstrating
the impact of building a general-purpose reasoning unit on complex domains such
as VideoQA.

通过介绍一种条件关系网络的方法来构建可复用神经元，用于视频中的表示和推理，并在 Well-known 的数据集上取得最新的状态技术，证明 CRN 对于视频问答等复杂域的表示和推理是成功的。