Building multisensory AI systems that learn from multiple sensory inputs such
as text, speech, video, real-world sensors, wearable devices, and medical data
holds great promise for impact in many scientific areas with practical
benefits, such as in supporting human health and well-being, enabling
multimedia content processing, and enhancing real-world autonomous agents. By
synthesizing a range of theoretical frameworks and application domains, this
thesis aims to advance the machine learning foundations of multisensory AI. In
the first part, we present a theoretical framework formalizing how modalities
interact with each other to give rise to new information for a task. These
interactions are the basic building blocks in all multimodal problems, and
their quantification enables users to understand their multimodal datasets,
design principled approaches to learn these interactions, and analyze whether
their model has succeeded in learning. In the second part, we study the design
of practical multimodal foundation models that generalize over many modalities
and tasks, which presents a step toward grounding large language models to
real-world sensory modalities. We introduce MultiBench, a unified large-scale
benchmark across a wide range of modalities, tasks, and research areas,
followed by the cross-modal attention and multimodal transformer architectures
that now underpin many of today's multimodal foundation models. Scaling these
architectures on MultiBench enables the creation of general-purpose
multisensory AI systems, and we discuss our collaborative efforts in applying
these models for real-world impact in affective computing, mental health,
cancer prognosis, and robotics. Finally, we conclude this thesis by discussing
how future work can leverage these ideas toward more general, interactive, and
safe multisensory AI.

通过综合一系列理论框架和应用领域，该论文旨在推进多传感器人工智能的机器学习基础。它首先提出了一个理论框架，形式化描述不同感知模态之间的相互作用，以获取任务的新信息。其次，研究了可广泛泛化于多个模态和任务的实用多模态基础模型的设计。最后，讨论了如何利用这些思想开展未来工作，以实现更普遍、互动和安全的多传感器人工智能。

多感官人工智能的基础

Foundations of Multisensory Artificial Intelligence

Projected to impact 1.6 million people in the UK by 2040 and costing
{\pounds}25 billion annually, dementia presents a growing challenge to society.
This study, a pioneering effort to predict the translational potential of
dementia research using machine learning, hopes to address the slow translation
of fundamental discoveries into practical applications despite dementia's
significant societal and economic impact. We used the Dimensions database to
extract data from 43,091 UK dementia research publications between the years
1990-2023, specifically metadata (authors, publication year etc.), concepts
mentioned in the paper, and the paper abstract. To prepare the data for machine
learning we applied methods such as one hot encoding and/or word embeddings. We
trained a CatBoost Classifier to predict if a publication will be cited in a
future patent or clinical trial. We trained several model variations. The model
combining metadata, concept, and abstract embeddings yielded the highest
performance: for patent predictions, an Area Under the Receiver Operating
Characteristic Curve (AUROC) of 0.84 and 77.17% accuracy; for clinical trial
predictions, an AUROC of 0.81 and 75.11% accuracy. The results demonstrate that
integrating machine learning within current research methodologies can uncover
overlooked publications, expediting the identification of promising research
and potentially transforming dementia research by predicting real-world impact
and guiding translational strategies.

使用机器学习技术，借助 Dimensions 数据库中的数据，研究人员对 1990-2023 年间英国痴呆研究出版物进行了分析，发现结合元数据、概念和摘要的模型具有最佳性能，证明将机器学习与当前研究方法相结合可以发现被忽视的出版物，加速找出有潜力的研究，并通过预测现实影响和指导翻译策略来改变痴呆研究。

机器学习促进转化医学研究：预测痴呆研究中的专利和临床试验纳入

Machine Learning to Promote Translational Research: Predicting Patent  and Clinical Trial Inclusion in Dementia Research

Learning meaningful representations that disentangle the underlying structure
of the data generating process is considered to be of key importance in machine
learning. While disentangled representations were found to be useful for
diverse tasks such as abstract reasoning and fair classification, their
scalability and real-world impact remain questionable. We introduce a new
high-resolution dataset with 1M simulated images and over 1,800 annotated
real-world images of the same setup. In contrast to previous work, this new
dataset exhibits correlations, a complex underlying structure, and allows to
evaluate transfer to unseen simulated and real-world settings where the encoder
i) remains in distribution or ii) is out of distribution. We propose new
architectures in order to scale disentangled representation learning to
realistic high-resolution settings and conduct a large-scale empirical study of
disentangled representations on this dataset. We observe that disentanglement
is a good predictor for out-of-distribution (OOD) task performance.

本文提出了一个新的高分辨率数据集，其中包括 1M 个模拟图像和 1800 个带注释的现实世界图像，以评估分解表示学习到未见过的模拟和实际环境中的泛化能力，研究表明分解表示学习是识别未知分布下任务性能的有效预测因素。