In recent years, Deep Learning has been successfully applied to multimodal
learning problems, with the aim of learning useful joint representations in
data fusion applications. When the available modalities consist of time series
data such as video, audio and sensor signals, it becomes imperative to consider
their temporal structure during the fusion process. In this paper, we propose
the Correlational Recurrent Neural Network (CorrRNN), a novel temporal fusion
model for fusing multiple input modalities that are inherently temporal in
nature. Key features of our proposed model include: (i) simultaneous learning
of the joint representation and temporal dependencies between modalities, (ii)
use of multiple loss terms in the objective function, including a maximum
correlation loss term to enhance learning of cross-modal information, and (iii)
the use of an attention model to dynamically adjust the contribution of
different input modalities to the joint representation. We validate our model
via experimentation on two different tasks: video- and sensor-based activity
classification, and audio-visual speech recognition. We empirically analyze the
contributions of different components of the proposed CorrRNN model, and
demonstrate its robustness, effectiveness and state-of-the-art performance on
multiple datasets.

本文介绍了一种名为 Correlational Recurrent Neural Network（CorrRNN）的神经网络模型，该模型将多种时间序列数据进行合并，通过同时学习联合表示和模态之间的时间依赖关系来优化预测性能，并使用多个损失项和注意力模型来调整模态之间的贡献。实验证明了该模型在各种数据集上表现优越。