The inherent challenge of multimodal fusion is to precisely capture the
cross-modal correlation and flexibly conduct cross-modal interaction. To fully
release the value of each modality and mitigate the influence of low-quality
multimodal data, dynamic multimodal fusion emerges as a promising learning
paradigm. Despite its widespread use, theoretical justifications in this field
are still notably lacking. Can we design a provably robust multimodal fusion
method? This paper provides theoretical understandings to answer this question
under a most popular multimodal fusion framework from the generalization
perspective. We proceed to reveal that several uncertainty estimation solutions
are naturally available to achieve robust multimodal fusion. Then a novel
multimodal fusion framework termed Quality-aware Multimodal Fusion (QMF) is
proposed, which can improve the performance in terms of classification accuracy
and model robustness. Extensive experimental results on multiple benchmarks can
support our findings.

本研究提供了一种新的质量感知多模态融合方法，旨在从概率推理的角度解决跨模态相关性和交互性的挑战，并在多个基准测试中验证了其分类准确性和模型鲁棒性的提升。

可证明的低质量多模态数据动态融合

Provable Dynamic Fusion for Low-Quality Multimodal Data

Deep multimodal learning has achieved great progress in recent years.
However, current fusion approaches are static in nature, i.e., they process and
fuse multimodal inputs with identical computation, without accounting for
diverse computational demands of different multimodal data. In this work, we
propose dynamic multimodal fusion (DynMM), a new approach that adaptively fuses
multimodal data and generates data-dependent forward paths during inference.
DynMM can reduce redundant computations for "easy" multimodal inputs (that can
be predicted correctly using only one modality or simple fusion techniques) and
retain representation power for "hard" samples by adopting all modalities and
complex fusion operations for prediction. Results on various multimodal tasks
demonstrate the efficiency and wide applicability of our approach. For
instance, DynMM can reduce the computation cost by 46.5% with a negligible
accuracy loss on CMU-MOSEI sentiment analysis. For RGB-D semantic segmentation
on NYU Depth data, DynMM achieves a +0.7% mIoU improvement with over 21%
reductions for the depth encoder when compared with strong baselines. We
believe this opens a novel direction towards dynamic multimodal network design,
with applications to a wide range of multimodal tasks.

本文提出了一种动态多模态融合的方法，可以在预测过程中根据数据的不同需求自适应地融合多模态数据，从而有效地减少计算成本，并在多个多模态任务上获得了良好的效果，这为动态多模态网络设计开辟了一条新的方向。