Recent breakthroughs in AI are poised to fundamentally enhance our study and
understanding of healthcare. The development of an integrated many-to-many
framework that leverages multiple data modality inputs for the analytical
modeling of multiple medical tasks, is critical for a unified understanding of
modern medicine. In this work, we introduce M3H, an explainable Multimodal
Multitask Machine Learning for Healthcare framework that consolidates learning
from diverse multimodal inputs across a broad spectrum of medical task
categories and machine learning problem classes. The modular design of the
framework ensures its generalizable data processing, task definition, and rapid
model prototyping, applicable to both clinical and operational healthcare
settings. We evaluate the M3H framework by validating models trained from four
modalities (tabular, time-series, language, and vision) on 41 medical tasks
across 4 machine learning problem classes. Our results demonstrate that M3H
consistently produces multitask models that outperform canonical single-task
models (by 1.1- 37.2%) across 37 disease diagnoses from 16 medical departments,
three hospital operation forecasts, and one patient phenotyping task: spanning
ML problem classes of supervised binary classification, multiclass
classification, regression, and clustering. Additionally, the framework
introduces a novel attention mechanism to balance self-exploitation (focus on
learning source task), and cross-exploration (encourage learning from other
tasks). Furthermore, M3H provides explainability insights on how joint learning
of additional tasks impacts the learning of source task using a proposed TIM
score, shedding light into the dynamics of task interdependencies. Its
adaptable architecture facilitates the customization and integration,
establishing it as a robust and scalable candidate solution for future
AI-driven healthcare systems.

AI 在医疗领域取得的最新突破有望从根本上提升我们对医疗的研究和理解。在这项工作中，我们介绍了 M3H 框架，它是一个可解释的多模态多任务医疗机器学习框架，能够从不同的多模态输入中学习，适用于各种医疗任务和机器学习问题类别，同时提供了有关任务相互依赖性动态的解释洞察。