Using multiple input streams simultaneously in training multimodal neural
networks is intuitively advantageous, but practically challenging. A key
challenge is unimodal bias, where a network overly relies on one modality and
ignores others during joint training. While unimodal bias is well-documented
empirically, our theoretical understanding of how architecture and data
statistics influence this bias remains incomplete. Here we develop a theory of
unimodal bias with deep multimodal linear networks. We calculate the duration
of the unimodal phase in learning as a function of the depth at which
modalities are fused within the network, dataset statistics, and
initialization. We find that the deeper the layer at which fusion occurs, the
longer the unimodal phase. A long unimodal phase can lead to a generalization
deficit and permanent unimodal bias in the overparametrized regime. In
addition, our theory reveals the modality learned first is not necessarily the
modality that contributes more to the output. Our results, derived for
multimodal linear networks, extend to ReLU networks in certain settings. Taken
together, this work illuminates pathologies of multimodal learning under joint
training, showing that late and intermediate fusion architectures can give rise
to long unimodal phases and permanent unimodal bias.

使用多输入流同时训练多模态神经网络在直觉上有优势，但在实践中具有挑战性。本文通过深度多模态线性网络的理论研究，揭示了联合训练中的单模态偏差问题，包括它与网络结构、数据统计和初始化的关系，以及它可能导致的泛化缺陷和永久性单模态偏差。此外，研究结果还表明，首先学习的模态未必对输出起更大的贡献。

多模态学习中的单模态偏差理论

A Theory of Unimodal Bias in Multimodal Learning

We tackle the problem of finding good architectures for multimodal
classification problems. We propose a novel and generic search space that spans
a large number of possible fusion architectures. In order to find an optimal
architecture for a given dataset in the proposed search space, we leverage an
efficient sequential model-based exploration approach that is tailored for the
problem. We demonstrate the value of posing multimodal fusion as a neural
architecture search problem by extensive experimentation on a toy dataset and
two other real multimodal datasets. We discover fusion architectures that
exhibit state-of-the-art performance for problems with different domain and
dataset size, including the NTU RGB+D dataset, the largest multi-modal action
recognition dataset available.

本文提出一种新颖的多模态融合架构搜索空间，使用序列模型搜索方法在不同领域和数据集中寻找优化架构，发现表现优异的融合架构，解决了多模态分类问题。