In this paper, we propose a novel approach (called GPT4MIA) that utilizes Generative Pre-trained Transformer (GPT) as a plug-and-play transductive inference tool for medical image analysis (MIA). We provide theoretical analysis on why a large pre-trained language model such as GPT-3 can be used as a plug-and-play transductive inference model for MIA. At the methodological level, we develop several technical treatments to improve the efficiency and effectiveness of GPT4MIA, including better prompt structure design, sample selection, and prompt ordering of representative samples/features. We present two concrete use cases (with workflow) of GPT4MIA: (1) detecting prediction errors and (2) improving prediction accuracy, working in conjecture with well-established vision-based models for image classification (e.g., ResNet). Experiments validate that our proposed method is effective for these two tasks. We further discuss the opportunities and challenges in utilizing Transformer-based large language models for broader MIA applications.

本研究提出了一种新的方法（称为GPT4MIA），利用生成预训练变压器（GPT）作为医学图像分析（MIA）的即插即用传递推理工具，在理论分析上，我们开发了多个技术处理，以提高GPT4MIA的效率和有效性，包括更好的提示结构设计、样本选择和代表性样本/特征的提示排序，并提供了两个具体的用例（带工作流程），以在与图像分类的成熟模型（例如ResNet）协同工作的情况下检测预测错误并提高预测准确度。

使用预训练的生成转换器（GPT-3）作为即插即用的感应模型用于医学图像分析的GPT4MIA