Recent advancements in Large Multimodal Models (LMMs) have attracted interest
in their generalization capability with only a few samples in the prompt. This
progress is particularly relevant to the medical domain, where the quality and
sensitivity of data pose unique challenges for model training and application.
However, the dependency on high-quality data for effective in-context learning
raises questions about the feasibility of these models when encountering with
the inevitable variations and errors inherent in real-world medical data. In
this paper, we introduce MID-M, a novel framework that leverages the in-context
learning capabilities of a general-domain Large Language Model (LLM) to process
multimodal data via image descriptions. MID-M achieves a comparable or superior
performance to task-specific fine-tuned LMMs and other general-domain ones,
without the extensive domain-specific training or pre-training on multimodal
data, with significantly fewer parameters. This highlights the potential of
leveraging general-domain LLMs for domain-specific tasks and offers a
sustainable and cost-effective alternative to traditional LMM developments.
Moreover, the robustness of MID-M against data quality issues demonstrates its
practical utility in real-world medical domain applications.

在本文中，我们介绍了 MID-M 这一创新框架，利用通用领域的大规模语言模型（LLM）的上下文学习能力，通过图像描述来处理多模态数据，其在性能上与任务特定的精调 LLMs 和其他通用领域的 LLMs 相当或更为出色，无需广泛域特定训练或多模态数据的预训练，参数更少，突出了利用通用领域 LLMs 进行领域特定任务的潜力，并为传统 LMM 发展提供了可持续和经济高效的替代方案。此外，MID-M 对于数据质量问题的稳健性证明了其在实际医学领域应用中的实用性。