Using multiple input streams simultaneously in training multimodal neural
networks is intuitively advantageous, but practically challenging. A key
challenge is unimodal bias, where a network overly relies on one modality and
ignores others during joint training. While unimodal bias is well-documented
empirically, our theoretical understanding of how architecture and data
statistics influence this bias remains incomplete. Here we develop a theory of
unimodal bias with deep multimodal linear networks. We calculate the duration
of the unimodal phase in learning as a function of the depth at which
modalities are fused within the network, dataset statistics, and
initialization. We find that the deeper the layer at which fusion occurs, the
longer the unimodal phase. A long unimodal phase can lead to a generalization
deficit and permanent unimodal bias in the overparametrized regime. In
addition, our theory reveals the modality learned first is not necessarily the
modality that contributes more to the output. Our results, derived for
multimodal linear networks, extend to ReLU networks in certain settings. Taken
together, this work illuminates pathologies of multimodal learning under joint
training, showing that late and intermediate fusion architectures can give rise
to long unimodal phases and permanent unimodal bias.

使用多输入流同时训练多模态神经网络在直觉上有优势，但在实践中具有挑战性。本文通过深度多模态线性网络的理论研究，揭示了联合训练中的单模态偏差问题，包括它与网络结构、数据统计和初始化的关系，以及它可能导致的泛化缺陷和永久性单模态偏差。此外，研究结果还表明，首先学习的模态未必对输出起更大的贡献。