Learning holistic computational representations in physical, chemical or
biological systems requires the ability to process information from different
distributions and modalities within the same model. Thus, the demand for
multimodal machine learning models has sharply risen for modalities that go
beyond vision and language, such as sequences, graphs, time series, or tabular
data. While there are many available multimodal fusion and alignment
approaches, most of them require end-to-end training, scale quadratically with
the number of modalities, cannot handle cases of high modality imbalance in the
training set, or are highly topology-specific, making them too restrictive for
many biomedical learning tasks. This paper presents Multimodal Lego (MM-Lego),
a modular and general-purpose fusion and model merging framework to turn any
set of encoders into a competitive multimodal model with no or minimal
fine-tuning. We achieve this by introducing a wrapper for unimodal encoders
that enforces lightweight dimensionality assumptions between modalities and
harmonises their representations by learning features in the frequency domain
to enable model merging with little signal interference. We show that MM-Lego
1) can be used as a model merging method which achieves competitive performance
with end-to-end fusion models without any fine-tuning, 2) can operate on any
unimodal encoder, and 3) is a model fusion method that, with minimal
fine-tuning, achieves state-of-the-art results on six benchmarked multimodal
biomedical tasks.

本研究提出了多模态 Lego（MM-Lego）框架，它是一个模块化和通用的融合和模型合并方法，可以将任何一组编码器转化为具有竞争力的多模态模型，不需要或仅需要进行最小的微调。通过引入对单模态编码器的包装，MM-Lego 强制实施模态之间的轻量级维度假设，并通过在频域学习特征来协调它们的表示，以使模型合并时干扰最小。实验证明，MM-Lego 可以作为模型合并的方法，达到了与端到端融合模型相媲美的性能，而无需进行任何微调；它可以在任何单模态编码器上运行；并且通过最小的微调，可以在六个基准多模态生物医学任务上实现最先进的结果。

MM-Lego：具有最小微调的模块化生物医学多模态模型

MM-Lego: Modular Biomedical Multimodal Models with Minimal Fine-Tuning

Despite an exciting new wave of multimodal machine learning models, current
approaches still struggle to interpret the complex contextual relationships
between the different modalities present in videos. Going beyond existing
methods that emphasize simple activities or objects, we propose a new
model-agnostic approach for generating detailed textual descriptions that
captures multimodal video information. Our method leverages the extensive
knowledge learnt by large language models, such as GPT-3.5 or Llama2, to reason
about textual descriptions of the visual and aural modalities, obtained from
BLIP-2, Whisper and ImageBind. Without needing additional finetuning of
video-text models or datasets, we demonstrate that available LLMs have the
ability to use these multimodal textual descriptions as proxies for ``sight''
or ``hearing'' and perform zero-shot multimodal classification of videos
in-context. Our evaluations on popular action recognition benchmarks, such as
UCF-101 or Kinetics, show these context-rich descriptions can be successfully
used in video understanding tasks. This method points towards a promising new
research direction in multimodal classification, demonstrating how an interplay
between textual, visual and auditory machine learning models can enable more
holistic video understanding.

通过利用大型语言模型（如 GPT-3.5 或 Llama2）的广泛知识，结合 BLIP-2、Whisper 和 ImageBind 获取的视觉和听觉多模态文本描述，我们提出了一种新的模型不可知方法，用于生成捕捉多模态视频信息的详细文本描述。我们的评估结果表明，这种基于文本描述的方法在视频理解任务中取得了成功，为多模态分类提供了一个有前景的新研究方向。