In this paper, we propose a novel text promptable surgical instrument
segmentation approach to overcome challenges associated with diversity and
differentiation of surgical instruments in minimally invasive surgeries. We
redefine the task as text promptable, thereby enabling a more nuanced
comprehension of surgical instruments and adaptability to new instrument types.
Inspired by recent advancements in vision-language models, we leverage
pretrained image and text encoders as our model backbone and design a text
promptable mask decoder consisting of attention- and convolution-based
prompting schemes for surgical instrument segmentation prediction. Our model
leverages multiple text prompts for each surgical instrument through a new
mixture of prompts mechanism, resulting in enhanced segmentation performance.
Additionally, we introduce a hard instrument area reinforcement module to
improve image feature comprehension and segmentation precision. Extensive
experiments on EndoVis2017 and EndoVis2018 datasets demonstrate our model's
superior performance and promising generalization capability. To our knowledge,
this is the first implementation of a promptable approach to surgical
instrument segmentation, offering significant potential for practical
application in the field of robotic-assisted surgery.

该研究提出了一种新颖的文本可提示的手术器械分割方法，利用预训练的图像和文本编码器作为模型骨干，其中包含注意力和卷积的提示方案，以及增强图像特征的部分，以提高手术器械分割性能。