Medical image classification requires labeled, task-specific datasets which
are used to train deep learning networks de novo, or to fine-tune foundation
models. However, this process is computationally and technically demanding. In
language processing, in-context learning provides an alternative, where models
learn from within prompts, bypassing the need for parameter updates. Yet,
in-context learning remains underexplored in medical image analysis. Here, we
systematically evaluate the model Generative Pretrained Transformer 4 with
Vision capabilities (GPT-4V) on cancer image processing with in-context
learning on three cancer histopathology tasks of high importance:
Classification of tissue subtypes in colorectal cancer, colon polyp subtyping
and breast tumor detection in lymph node sections. Our results show that
in-context learning is sufficient to match or even outperform specialized
neural networks trained for particular tasks, while only requiring a minimal
number of samples. In summary, this study demonstrates that large vision
language models trained on non-domain specific data can be applied out-of-the
box to solve medical image-processing tasks in histopathology. This
democratizes access of generalist AI models to medical experts without
technical background especially for areas where annotated data is scarce.

本研究评估了具有上下文学习能力的模型 Generative Pretrained Transformer 4 with Vision (GPT-4V) 在癌症图像处理中的应用，结果显示在组织亚型分类、结肠息肉亚型分类和淋巴结切片中的乳腺肿瘤检测等三项癌症组织病理学任务中，使用上下文学习可以达到或超过针对特定任务训练的神经网络的性能，且只需使用较少的样本。总结起来，本研究表明，在病理学中，基于非领域特定数据训练的大型视觉语言模型可以直接应用于解决医学图像处理任务，这使得非技术背景的医学专家在样本稀缺的领域中能够使用通用的人工智能模型。