Building multisensory AI systems that learn from multiple sensory inputs such
as text, speech, video, real-world sensors, wearable devices, and medical data
holds great promise for impact in many scientific areas with practical
benefits, such as in supporting human health and well-being, enabling
multimedia content processing, and enhancing real-world autonomous agents. By
synthesizing a range of theoretical frameworks and application domains, this
thesis aims to advance the machine learning foundations of multisensory AI. In
the first part, we present a theoretical framework formalizing how modalities
interact with each other to give rise to new information for a task. These
interactions are the basic building blocks in all multimodal problems, and
their quantification enables users to understand their multimodal datasets,
design principled approaches to learn these interactions, and analyze whether
their model has succeeded in learning. In the second part, we study the design
of practical multimodal foundation models that generalize over many modalities
and tasks, which presents a step toward grounding large language models to
real-world sensory modalities. We introduce MultiBench, a unified large-scale
benchmark across a wide range of modalities, tasks, and research areas,
followed by the cross-modal attention and multimodal transformer architectures
that now underpin many of today's multimodal foundation models. Scaling these
architectures on MultiBench enables the creation of general-purpose
multisensory AI systems, and we discuss our collaborative efforts in applying
these models for real-world impact in affective computing, mental health,
cancer prognosis, and robotics. Finally, we conclude this thesis by discussing
how future work can leverage these ideas toward more general, interactive, and
safe multisensory AI.

通过综合一系列理论框架和应用领域，该论文旨在推进多传感器人工智能的机器学习基础。它首先提出了一个理论框架，形式化描述不同感知模态之间的相互作用，以获取任务的新信息。其次，研究了可广泛泛化于多个模态和任务的实用多模态基础模型的设计。最后，讨论了如何利用这些思想开展未来工作，以实现更普遍、互动和安全的多传感器人工智能。