Human perception of the empirical world involves recognizing the diverse
appearances, or 'modalities', of underlying objects. Despite the longstanding
consideration of this perspective in philosophy and cognitive science, the
study of multimodality remains relatively under-explored within the field of
machine learning. Nevertheless, current studies of multimodal machine learning
are limited to empirical practices, lacking theoretical foundations beyond
heuristic arguments. An intriguing finding from the practice of multimodal
learning is that a model trained on multiple modalities can outperform a
finely-tuned unimodal model, even on unimodal tasks. This paper provides a
theoretical framework that explains this phenomenon, by studying generalization
properties of multimodal learning algorithms. We demonstrate that multimodal
learning allows for a superior generalization bound compared to unimodal
learning, up to a factor of $O(\sqrt{n})$, where $n$ represents the sample
size. Such advantage occurs when both connection and heterogeneity exist
between the modalities.

研究多模态学习算法的泛化性质，发现与单一模态学习相比，多模态学习可以达到更好的泛化界限，最多可以提高到 O (√n) 倍，其中 n 代表样本大小。