The Segment Anything Model (SAM) is a powerful vision foundation model that
is revolutionizing the traditional paradigm of segmentation. Despite this, a
reliance on prompting each frame and large computational cost limit its usage
in robotically assisted surgery. Applications, such as augmented reality
guidance, require little user intervention along with efficient inference to be
usable clinically. In this study, we address these limitations by adopting
lightweight SAM variants to meet the speed requirement and employing
fine-tuning techniques to enhance their generalization in surgical scenes.
Recent advancements in Tracking Any Point (TAP) have shown promising results in
both accuracy and efficiency, particularly when points are occluded or leave
the field of view. Inspired by this progress, we present a novel framework that
combines an online point tracker with a lightweight SAM model that is
fine-tuned for surgical instrument segmentation. Sparse points within the
region of interest are tracked and used to prompt SAM throughout the video
sequence, providing temporal consistency. The quantitative results surpass the
state-of-the-art semi-supervised video object segmentation method on the
EndoVis 2015 dataset, with an over 25 FPS inference speed running on a single
GeForce RTX 4060 GPU.

在这项研究中，我们采用轻量级的 Segment Anything Model (SAM) 变体以满足速度要求，并采用微调技术来增强其在外科手术场景中的泛化能力。我们结合在线点追踪器和经过微调的轻量级 SAM 模型构建了一个新颖的框架，用于外科器械分割。在 EndoVis 2015 数据集上，定量结果超过了半监督视频对象分割方法的最新水平，并且可以在单个 GeForce RTX 4060 GPU 上以超过 25 FPS 的推理速度运行。

视频中实时外科器械分割：基于点跟踪和全能分割

Real-time Surgical Instrument Segmentation in Video Using Point Tracking  and Segment Anything

Deformable tracking and real-time estimation of 3D tissue motion is essential
to enable automation and image guidance applications in robotically assisted
surgery. Our model, Sparse Efficient Neural Depth and Deformation (SENDD),
extends prior 2D tracking work to estimate flow in 3D space. SENDD introduces
novel contributions of learned detection, and sparse per-point depth and 3D
flow estimation, all with less than half a million parameters. SENDD does this
by using graph neural networks of sparse keypoint matches to estimate both
depth and 3D flow. We quantify and benchmark SENDD on a comprehensively
labelled tissue dataset, and compare it to an equivalent 2D flow model. SENDD
performs comparably while enabling applications that 2D flow cannot. SENDD can
track points and estimate depth at 10fps on an NVIDIA RTX 4000 for 1280 tracked
(query) points and its cost scales linearly with an increasing/decreasing
number of points. SENDD enables multiple downstream applications that require
3D motion estimation.

研究开发了一种基于稀疏图神经网络的 3D 组织运动的快速跟踪与实时估计算法 SENDD，该算法能够实现单点深度与 3D 流估计，具有较小的参数量和良好的跟踪能力，从而可以广泛应用在机器人辅助手术和影像导航等多个领域。