In this study, we uncover the unexpected efficacy of residual-based large
language models (LLMs) as part of encoders for biomedical imaging tasks, a
domain traditionally devoid of language or textual data. The approach diverges
from established methodologies by utilizing a frozen transformer block,
extracted from pre-trained LLMs, as an innovative encoder layer for the direct
processing of visual tokens. This strategy represents a significant departure
from the standard multi-modal vision-language frameworks, which typically hinge
on language-driven prompts and inputs. We found that these LLMs could boost
performance across a spectrum of biomedical imaging applications, including
both 2D and 3D visual classification tasks, serving as plug-and-play boosters.
More interestingly, as a byproduct, we found that the proposed framework
achieved superior performance, setting new state-of-the-art results on
extensive, standardized datasets in MedMNIST-2D and 3D. Through this work, we
aim to open new avenues for employing LLMs in biomedical imaging and enriching
the understanding of their potential in this specialized domain.

该研究揭示了残差型大型语言模型在生物医学图像任务中作为编码器的意外有效性，这在传统上与语言或文本数据无关。通过利用预训练的大型语言模型中的冻结变压器块作为创新编码器层，该方法与现有方法学不同，可以直接处理视觉标记。该研究发现，这些大型语言模型能够提升生物医学图像应用的性能，包括 2D 和 3D 的视觉分类任务，作为即插即用的增强器。更有趣的是，该提出的框架在 MedMNIST-2D 和 3D 的大规模标准数据集上取得了卓越的性能，刷新了最新的技术结果。通过这项工作，我们的目标是在生物医学图像领域开拓大型语言模型的应用新途径，并丰富对其在这一专业领域的潜力的认识。