Intelligently reasoning about the world often requires integrating data from
multiple modalities, as any individual modality may contain unreliable or
incomplete information. Prior work in multimodal learning fuses input
modalities only after significant independent processing. On the other hand,
the brain performs multimodal processing almost immediately. This divide
between conventional multimodal learning and neuroscience suggests that a
detailed study of early multimodal fusion could improve artificial multimodal
representations. To facilitate the study of early multimodal fusion, we create
a convolutional LSTM network architecture that simultaneously processes both
audio and visual inputs, and allows us to select the layer at which audio and
visual information combines. Our results demonstrate that immediate fusion of
audio and visual inputs in the initial C-LSTM layer results in higher
performing networks that are more robust to the addition of white noise in both
audio and visual inputs.

该篇论文通过创建卷积 LSTM 网络结构，研究了多模态表示学习中视听融合的早期处理，结果表明，在初始 C-LSTM 层中立即融合音频和视觉输入可以提高网络的性能，使其更能够抵抗白噪声的干扰。

多模态表示学习中早期融合的好处

On the Benefits of Early Fusion in Multimodal Representation Learning

In this work, we present a machine learning approach for reducing the error
when numerically solving time-dependent partial differential equations (PDE).
We use a fully convolutional LSTM network to exploit the spatiotemporal
dynamics of PDEs. The neural network serves to enhance finite-difference and
finite-volume methods (FDM/FVM) that are commonly used to solve PDEs, allowing
us to maintain guarantees on the order of convergence of our method. We train
the network on simulation data, and show that our network can reduce error by a
factor of 2 to 3 compared to the baseline algorithms. We demonstrate our method
on three PDEs that each feature qualitatively different dynamics. We look at
the linear advection equation, which propagates its initial conditions at a
constant speed, the inviscid Burgers' equation, which develops shockwaves, and
the Kuramoto-Sivashinsky (KS) equation, which is chaotic.

本文介绍了一种利用机器学习方法来减少时间依赖性偏微分方程数值求解误差的方法，使用全卷积 LSTM 网络来利用偏微分方程的时空动态性，提高常规有限差分和有限体积法的精度，并通过对模拟数据的训练和三个不同动态学特征的偏微分方程实例的演示，表明该方法与其他算法相比误差降低了 2 到 3 倍。