Despite significant advancements in robotic systems and surgical data
science, ensuring safe and optimal execution in robot-assisted minimally
invasive surgery (RMIS) remains a complex challenge. Current surgical error
detection methods involve two parts: identifying surgical gestures and then
detecting errors within each gesture clip. These methods seldom consider the
rich contextual and semantic information inherent in surgical videos, limiting
their performance due to reliance on accurate gesture identification. Motivated
by the chain-of-thought prompting in natural language processing, this letter
presents a novel and real-time end-to-end error detection framework,
Chain-of-Thought (COG) prompting, leveraging contextual information from
surgical videos. This encompasses two reasoning modules designed to mimic the
decision-making processes of expert surgeons. Concretely, we first design a
Gestural-Visual Reasoning module, which utilizes transformer and attention
architectures for gesture prompting, while the second, a Multi-Scale Temporal
Reasoning module, employs a multi-stage temporal convolutional network with
both slow and fast paths for temporal information extraction. We extensively
validate our method on the public benchmark RMIS dataset JIGSAWS. Our method
encapsulates the reasoning processes inherent to surgical activities enabling
it to outperform the state-of-the-art by 4.6% in F1 score, 4.6% in Accuracy,
and 5.9% in Jaccard index while processing each frame in 6.69 milliseconds on
average, demonstrating the great potential of our approach in enhancing the
safety and efficacy of RMIS procedures and surgical education. The code will be
available.

本论文提出了一种新颖的实时端到端错误检测框架，称为 Chain-of-Thought (COG) prompting，利用手术视频中的上下文信息来提高机器人辅助微创手术的安全性和效果。通过使用 transformer、attention 和多阶段时序卷积网络等方法，该方法在公共基准 RMIS 数据集 JIGSAWS 上优于目前最先进方法的 F1 分数、准确率和 Jaccard 指数，并且平均每帧处理时间为 6.69 毫秒。

逐步思考：机器人手术视频中错误检测的连续手势提示

Think Step by Step: Chain-of-Gesture Prompting for Error Detection in  Robotic Surgical Videos

Robot-assisted minimally invasive surgery benefits from enhancing dynamic
scene reconstruction, as it improves surgical outcomes. While Neural Radiance
Fields (NeRF) have been effective in scene reconstruction, their slow inference
speeds and lengthy training durations limit their applicability. To overcome
these limitations, 3D Gaussian Splatting (3D-GS) based methods have emerged as
a recent trend, offering rapid inference capabilities and superior 3D quality.
However, these methods still struggle with under-reconstruction in both static
and dynamic scenes. In this paper, we propose HFGS, a novel approach for
deformable endoscopic reconstruction that addresses these challenges from
spatial and temporal frequency perspectives. Our approach incorporates
deformation fields to better handle dynamic scenes and introduces Spatial
High-Frequency Emphasis Reconstruction (SHF) to minimize discrepancies in
spatial frequency spectra between the rendered image and its ground truth.
Additionally, we introduce Temporal High-Frequency Emphasis Reconstruction
(THF) to enhance dynamic awareness in neural rendering by leveraging flow
priors, focusing optimization on motion-intensive parts. Extensive experiments
on two widely used benchmarks demonstrate that HFGS achieves superior rendering
quality. Our code will be available.

提出了一种名为 HFGS 的新方法，通过空间和时间频率角度处理动态场景的可变形内窥镜重建问题，并引入了空间高频重建和时间高频重建技术来提高神经渲染的动态感知和 3D 质量。

HFGS: 四维高斯完全投影，着重于内窥镜场景重建中的空间和时间高频分量

HFGS: 4D Gaussian Splatting with Emphasis on Spatial and Temporal  High-Frequency Components for Endoscopic Scene Reconstruction

In the realm of robot-assisted minimally invasive surgery, dynamic scene
reconstruction can significantly enhance downstream tasks and improve surgical
outcomes. Neural Radiance Fields (NeRF)-based methods have recently risen to
prominence for their exceptional ability to reconstruct scenes. Nonetheless,
these methods are hampered by slow inference, prolonged training, and
substantial computational demands. Additionally, some rely on stereo depth
estimation, which is often infeasible due to the high costs and logistical
challenges associated with stereo cameras. Moreover, the monocular
reconstruction quality for deformable scenes is currently inadequate. To
overcome these obstacles, we present Endo-4DGS, an innovative, real-time
endoscopic dynamic reconstruction approach that utilizes 4D Gaussian Splatting
(GS) and requires no ground truth depth data. This method extends 3D GS by
incorporating a temporal component and leverages a lightweight MLP to capture
temporal Gaussian deformations. This effectively facilitates the reconstruction
of dynamic surgical scenes with variable conditions. We also integrate
Depth-Anything to generate pseudo-depth maps from monocular views, enhancing
the depth-guided reconstruction process. Our approach has been validated on two
surgical datasets, where it has proven to render in real-time, compute
efficiently, and reconstruct with remarkable accuracy. These results underline
the vast potential of Endo-4DGS to improve surgical assistance.

在机器人辅助微创手术领域，动态场景重建可以显著提高下游任务和改善手术结果。我们提出了一种创新的实时内窥镜动态重建方法 Endo-4DGS，使用了 4D 高斯飞溅和无需地面真实深度数据。该方法通过引入时间组件，借助轻量级 MLP 捕捉时间高斯变形，有效实现了具有可变条件的动态手术场景的重建。我们还整合了 Depth-Anything，从单目视图中生成伪深度图，增强了深度引导重建过程。我们的方法在两个手术数据集上通过验证，证明了实时渲染、高效计算和重建准确性，彰显了 Endo-4DGS 改善手术辅助的巨大潜力。