Learning holistic computational representations in physical, chemical or
biological systems requires the ability to process information from different
distributions and modalities within the same model. Thus, the demand for
multimodal machine learning models has sharply risen for modalities that go
beyond vision and language, such as sequences, graphs, time series, or tabular
data. While there are many available multimodal fusion and alignment
approaches, most of them require end-to-end training, scale quadratically with
the number of modalities, cannot handle cases of high modality imbalance in the
training set, or are highly topology-specific, making them too restrictive for
many biomedical learning tasks. This paper presents Multimodal Lego (MM-Lego),
a modular and general-purpose fusion and model merging framework to turn any
set of encoders into a competitive multimodal model with no or minimal
fine-tuning. We achieve this by introducing a wrapper for unimodal encoders
that enforces lightweight dimensionality assumptions between modalities and
harmonises their representations by learning features in the frequency domain
to enable model merging with little signal interference. We show that MM-Lego
1) can be used as a model merging method which achieves competitive performance
with end-to-end fusion models without any fine-tuning, 2) can operate on any
unimodal encoder, and 3) is a model fusion method that, with minimal
fine-tuning, achieves state-of-the-art results on six benchmarked multimodal
biomedical tasks.

本研究提出了多模态 Lego（MM-Lego）框架，它是一个模块化和通用的融合和模型合并方法，可以将任何一组编码器转化为具有竞争力的多模态模型，不需要或仅需要进行最小的微调。通过引入对单模态编码器的包装，MM-Lego 强制实施模态之间的轻量级维度假设，并通过在频域学习特征来协调它们的表示，以使模型合并时干扰最小。实验证明，MM-Lego 可以作为模型合并的方法，达到了与端到端融合模型相媲美的性能，而无需进行任何微调；它可以在任何单模态编码器上运行；并且通过最小的微调，可以在六个基准多模态生物医学任务上实现最先进的结果。