Deep multimodal learning has achieved great progress in recent years.
However, current fusion approaches are static in nature, i.e., they process and
fuse multimodal inputs with identical computation, without accounting for
diverse computational demands of different multimodal data. In this work, we
propose dynamic multimodal fusion (DynMM), a new approach that adaptively fuses
multimodal data and generates data-dependent forward paths during inference.
DynMM can reduce redundant computations for "easy" multimodal inputs (that can
be predicted correctly using only one modality or simple fusion techniques) and
retain representation power for "hard" samples by adopting all modalities and
complex fusion operations for prediction. Results on various multimodal tasks
demonstrate the efficiency and wide applicability of our approach. For
instance, DynMM can reduce the computation cost by 46.5% with a negligible
accuracy loss on CMU-MOSEI sentiment analysis. For RGB-D semantic segmentation
on NYU Depth data, DynMM achieves a +0.7% mIoU improvement with over 21%
reductions for the depth encoder when compared with strong baselines. We
believe this opens a novel direction towards dynamic multimodal network design,
with applications to a wide range of multimodal tasks.

本文提出了一种动态多模态融合的方法，可以在预测过程中根据数据的不同需求自适应地融合多模态数据，从而有效地减少计算成本，并在多个多模态任务上获得了良好的效果，这为动态多模态网络设计开辟了一条新的方向。