Multimodal machine learning is a vibrant multi-disciplinary research field
that aims to design computer agents with intelligent capabilities such as
understanding, reasoning, and learning through integrating multiple
communicative modalities, including linguistic, acoustic, visual, tactile, and
physiological messages. With the recent interest in video understanding,
embodied autonomous agents, text-to-image generation, and multisensor fusion in
application domains such as healthcare and robotics, multimodal machine
learning has brought unique computational and theoretical challenges to the
machine learning community given the heterogeneity of data sources and the
interconnections often found between modalities. However, the breadth of
progress in multimodal research has made it difficult to identify the common
themes and open questions in the field. By synthesizing a broad range of
application domains and theoretical frameworks from both historical and recent
perspectives, this paper is designed to provide an overview of the
computational and theoretical foundations of multimodal machine learning. We
start by defining three key principles of modality heterogeneity, connections,
and interactions that have driven subsequent innovations, and propose a
taxonomy of six core technical challenges: representation, alignment,
reasoning, generation, transference, and quantification covering historical and
recent trends. Recent technical achievements will be presented through the lens
of this taxonomy, allowing researchers to understand the similarities and
differences across new approaches. We end by motivating several open problems
for future research as identified by our taxonomy.

本文旨在探讨多模态机器学习的计算与理论基础，定义了三个关键原则和六个核心技术挑战，并提出多个未来研究的开放性问题。