Prompt-based segmentation, also known as interactive segmentation, has
recently become a popular approach in image segmentation. A well-designed
prompt-based model called Segment Anything Model (SAM) has demonstrated its
ability to segment a wide range of natural images, which has sparked a lot of
discussion in the community. However, recent studies have shown that SAM
performs poorly on medical images. This has motivated us to design a new
prompt-based segmentation model specifically for medical image segmentation. In
this paper, we combine the prompted-based segmentation paradigm with UNet,
which is a widly-recognized successful architecture for medical image
segmentation. We have named the resulting model PromptUNet. In order to adapt
the real-world clinical use, we expand the existing prompt types in SAM to
include novel Supportive Prompts and En-face Prompts. We have evaluated the
capabilities of PromptUNet on 19 medical image segmentation tasks using a
variety of image modalities, including CT, MRI, ultrasound, fundus, and
dermoscopic images. Our results show that PromptUNet outperforms a wide range
of state-of-the-art (SOTA) medical image segmentation methods, including
nnUNet, TransUNet, UNetr, MedSegDiff, and MSA. Code will be released at:
this https URL

本文针对现有 Prompt 模型在医学图像分割方面的表现较差的问题，将 Prompt-based segmentation 和 UNet 结合起来设计出一种新的医学图像分割模型 ——PromptUNet，并新添了 Supportive Prompts 和 En-face Prompts 等 Prompt 类型，实验结果表明 PromptUNet 在医学图像分割上的精度高于当前主流的 nnUNet、TransUNet、UNetr、MedSegDiff 和 MSA 等方法。

PromptUNet: 拟人化交互式医学图像分割

PromptUNet: Toward Interactive Medical Image Segmentation

Image segmentation is usually addressed by training a model for a fixed set
of object classes. Incorporating additional classes or more complex queries
later is expensive as it requires re-training the model on a dataset that
encompasses these expressions. Here we propose a system that can generate image
segmentations based on arbitrary prompts at test time. A prompt can be either a
text or an image. This approach enables us to create a unified model (trained
once) for three common segmentation tasks, which come with distinct challenges:
referring expression segmentation, zero-shot segmentation and one-shot
segmentation. We build upon the CLIP model as a backbone which we extend with a
transformer-based decoder that enables dense prediction. After training on an
extended version of the PhraseCut dataset, our system generates a binary
segmentation map for an image based on a free-text prompt or on an additional
image expressing the query. We analyze different variants of the latter
image-based prompts in detail. This novel hybrid input allows for dynamic
adaptation not only to the three segmentation tasks mentioned above, but to any
binary segmentation task where a text or image query can be formulated.
Finally, we find our system to adapt well to generalized queries involving
affordances or properties. Code is available at
this https URL

本文提出了一种基于文本或图像提示产生图像分割结果的模型，使用 CLIP 模型并引入 Transformer-based decoder 实现了对于任意提示的图像分割。经过对短语切割数据集进行扩展训练，该模型可以生成二进制的分割结果。我们发现该模型不仅可适应于三种标准的分割任务，而且可以适应于任何文本或图像提示的二进制分割任务。