Deep multimodal learning has achieved great progress in recent years.
However, current fusion approaches are static in nature, i.e., they process and
fuse multimodal inputs with identical computation, without accounting for
diverse computational demands of different multimodal data. In this work, we
propose dynamic multimodal fusion (DynMM), a new approach that adaptively fuses
multimodal data and generates data-dependent forward paths during inference.
DynMM can reduce redundant computations for "easy" multimodal inputs (that can
be predicted correctly using only one modality or simple fusion techniques) and
retain representation power for "hard" samples by adopting all modalities and
complex fusion operations for prediction. Results on various multimodal tasks
demonstrate the efficiency and wide applicability of our approach. For
instance, DynMM can reduce the computation cost by 46.5% with a negligible
accuracy loss on CMU-MOSEI sentiment analysis. For RGB-D semantic segmentation
on NYU Depth data, DynMM achieves a +0.7% mIoU improvement with over 21%
reductions for the depth encoder when compared with strong baselines. We
believe this opens a novel direction towards dynamic multimodal network design,
with applications to a wide range of multimodal tasks.

本文提出了一种动态多模态融合的方法，可以在预测过程中根据数据的不同需求自适应地融合多模态数据，从而有效地减少计算成本，并在多个多模态任务上获得了良好的效果，这为动态多模态网络设计开辟了一条新的方向。

动态多模态融合

Dynamic Multimodal Fusion

In this paper, we present methods in deep multimodal learning for fusing
speech and visual modalities for Audio-Visual Automatic Speech Recognition
(AV-ASR). First, we study an approach where uni-modal deep networks are trained
separately and their final hidden layers fused to obtain a joint feature space
in which another deep network is built. While the audio network alone achieves
a phone error rate (PER) of $41\%$ under clean condition on the IBM large
vocabulary audio-visual studio dataset, this fusion model achieves a PER of
$35.83\%$ demonstrating the tremendous value of the visual channel in phone
classification even in audio with high signal to noise ratio. Second, we
present a new deep network architecture that uses a bilinear softmax layer to
account for class specific correlations between modalities. We show that
combining the posteriors from the bilinear networks with those from the fused
model mentioned above results in a further significant phone error rate
reduction, yielding a final PER of $34.03\%$.

本文介绍深度多模态学习的方法，用于合并语音和视觉特征进行音视频自动语音识别。实验结果表明，使用深度网络的融合模型和双线性 softmax 层能够进一步降低电话错误率。