Multimodal deep learning has been used to predict clinical endpoints and
diagnoses from clinical routine data. However, these models suffer from scaling
issues: they have to learn pairwise interactions between each piece of
information in each data type, thereby escalating model complexity beyond
manageable scales. This has so far precluded a widespread use of multimodal
deep learning. Here, we present a new technical approach of "learnable
synergies", in which the model only selects relevant interactions between data
modalities and keeps an "internal memory" of relevant data. Our approach is
easily scalable and naturally adapts to multimodal data inputs from clinical
routine. We demonstrate this approach on three large multimodal datasets from
radiology and ophthalmology and show that it outperforms state-of-the-art
models in clinically relevant diagnosis tasks. Our new approach is transferable
and will allow the application of multimodal deep learning to a broad set of
clinically relevant problems.

提出了新的可学习的协同学习方法，用以解决多模态深度学习模型中所存在的模型复杂性升级问题，以及其应用于医学临床数据处理中所面临的扩展性难题，并在三个多模态数据集上展现了其卓越的表现。