Existing promptable segmentation methods in the medical imaging field
primarily consider either textual or visual prompts to segment relevant
objects, yet they often fall short when addressing anomalies in medical images,
like tumors, which may vary greatly in shape, size, and appearance. Recognizing
the complexity of medical scenarios and the limitations of textual or visual
prompts, we propose a novel dual-prompt schema that leverages the complementary
strengths of visual and textual prompts for segmenting various organs and
tumors. Specifically, we introduce CAT, an innovative model that Coordinates
Anatomical prompts derived from 3D cropped images with Textual prompts enriched
by medical domain knowledge. The model architecture adopts a general
query-based design, where prompt queries facilitate segmentation queries for
mask prediction. To synergize two types of prompts within a unified framework,
we implement a ShareRefiner, which refines both segmentation and prompt queries
while disentangling the two types of prompts. Trained on a consortium of 10
public CT datasets, CAT demonstrates superior performance in multiple
segmentation tasks. Further validation on a specialized in-house dataset
reveals the remarkable capacity of segmenting tumors across multiple cancer
stages. This approach confirms that coordinating multimodal prompts is a
promising avenue for addressing complex scenarios in the medical domain.

在医学影像领域，现有的可提示分割方法主要考虑文本或视觉提示，但在处理医学图像中的异常情况（如肿瘤）时往往存在一些局限性，因为肿瘤在形状、大小和外观等方面可能存在很大的变异。为了解决医学场景的复杂性和文本或视觉提示的局限性，我们提出了一种新颖的双提示模式，利用视觉和文本提示的互补优势来分割各种器官和肿瘤。具体而言，我们引入了一种创新的模型 CAT，它通过医学领域知识丰富的 3D 裁剪图像生成解剖提示，与富含医学领域知识的文本提示进行配合。该模型采用了基于查询的通用设计，其中提示查询有助于进行掩模预测的分割查询。为了在统一框架内协调两种类型的提示，我们实现了一个 ShareRefiner，它同时改进了分割和提示查询，同时解耦了两种类型的提示。在由 10 个公共 CT 数据集组成的联合培训集上进行训练，CAT 在多个分割任务中展现出卓越的性能。在一组特殊的内部数据集上进行的进一步验证显示了在多个癌症阶段分割肿瘤的显著能力。这种方法验证了协调多模式提示是解决医学领域复杂场景的一种有前途的途径。