This paper describes a network that captures multimodal correlations over
arbitrary timestamps. The proposed scheme operates as a complementary, extended
network over a multimodal convolutional neural network (CNN). Spatial and
temporal streams are required for action recognition by a deep CNN, but
overfitting reduction and fusing these two streams remain open problems. The
existing fusion approach averages the two streams. Here we propose a
correlation network with a Shannon fusion for learning a pre-trained CNN. A
Long-range video may consist of spatiotemporal correlations over arbitrary
times, which can be captured by forming the correlation network from simple
fully connected layers. This approach was found to complement the existing
network fusion methods. The importance of multimodal correlation is validated
in comparison experiments on the UCF-101 and HMDB-51 datasets. The multimodal
correlation enhanced the accuracy of the video recognition results.

本文介绍了一种网络，能够在任意时间戳上捕获多模态相关性，通过用莎侬融合法扩展了多模态卷积神经网络，提出了用相关网络学习预先训练的卷积神经网络的方法，并在 UCF-101 和 HMDB-51 数据集上进行实验，结果显示多模态相关性能够提高视频识别结果的准确性。