Despite significant advancements in robotic systems and surgical data
science, ensuring safe and optimal execution in robot-assisted minimally
invasive surgery (RMIS) remains a complex challenge. Current surgical error
detection methods involve two parts: identifying surgical gestures and then
detecting errors within each gesture clip. These methods seldom consider the
rich contextual and semantic information inherent in surgical videos, limiting
their performance due to reliance on accurate gesture identification. Motivated
by the chain-of-thought prompting in natural language processing, this letter
presents a novel and real-time end-to-end error detection framework,
Chain-of-Thought (COG) prompting, leveraging contextual information from
surgical videos. This encompasses two reasoning modules designed to mimic the
decision-making processes of expert surgeons. Concretely, we first design a
Gestural-Visual Reasoning module, which utilizes transformer and attention
architectures for gesture prompting, while the second, a Multi-Scale Temporal
Reasoning module, employs a multi-stage temporal convolutional network with
both slow and fast paths for temporal information extraction. We extensively
validate our method on the public benchmark RMIS dataset JIGSAWS. Our method
encapsulates the reasoning processes inherent to surgical activities enabling
it to outperform the state-of-the-art by 4.6% in F1 score, 4.6% in Accuracy,
and 5.9% in Jaccard index while processing each frame in 6.69 milliseconds on
average, demonstrating the great potential of our approach in enhancing the
safety and efficacy of RMIS procedures and surgical education. The code will be
available.

本论文提出了一种新颖的实时端到端错误检测框架，称为 Chain-of-Thought (COG) prompting，利用手术视频中的上下文信息来提高机器人辅助微创手术的安全性和效果。通过使用 transformer、attention 和多阶段时序卷积网络等方法，该方法在公共基准 RMIS 数据集 JIGSAWS 上优于目前最先进方法的 F1 分数、准确率和 Jaccard 指数，并且平均每帧处理时间为 6.69 毫秒。

逐步思考：机器人手术视频中错误检测的连续手势提示

Think Step by Step: Chain-of-Gesture Prompting for Error Detection in  Robotic Surgical Videos

The visual-question localized-answering (VQLA) system can serve as a
knowledgeable assistant in surgical education. Except for providing text-based
answers, the VQLA system can highlight the interested region for better
surgical scene understanding. However, deep neural networks (DNNs) suffer from
catastrophic forgetting when learning new knowledge. Specifically, when DNNs
learn on incremental classes or tasks, their performance on old tasks drops
dramatically. Furthermore, due to medical data privacy and licensing issues, it
is often difficult to access old data when updating continual learning (CL)
models. Therefore, we develop a non-exemplar continual surgical VQLA framework,
to explore and balance the rigidity-plasticity trade-off of DNNs in a
sequential learning paradigm. We revisit the distillation loss in CL tasks, and
propose rigidity-plasticity-aware distillation (RP-Dist) and self-calibrated
heterogeneous distillation (SH-Dist) to preserve the old knowledge. The weight
aligning (WA) technique is also integrated to adjust the weight bias between
old and new tasks. We further establish a CL framework on three public surgical
datasets in the context of surgical settings that consist of overlapping
classes between old and new surgical VQLA tasks. With extensive experiments, we
demonstrate that our proposed method excellently reconciles learning and
forgetting on the continual surgical VQLA over conventional CL methods. Our
code is publicly accessible.

通过开发非样本连续外科 VQLA 框架，我们在连续学习范式中探索和平衡深度神经网络的刚性 - 可塑性权衡，并提出了 RP-Dist 和 SH-Dist 来保留旧知识。在外科设置的三个公共数据集上建立了一个连续学习框架，通过广泛的实验证明我们的方法在连续外科 VQLA 上优于传统的连续学习方法。

再探视觉问答机器人手术中连续学习的蒸馏

Revisiting Distillation for Continual Learning on Visual Question  Localized-Answering in Robotic Surgery

Purpose: A profound education of novice surgeons is crucial to ensure that
surgical interventions are effective and safe. One important aspect is the
teaching of technical skills for minimally invasive or robot-assisted
procedures. This includes the objective and preferably automatic assessment of
surgical skill. Recent studies presented good results for automatic, objective
skill evaluation by collecting and analyzing motion data such as trajectories
of surgical instruments. However, obtaining the motion data generally requires
additional equipment for instrument tracking or the availability of a robotic
surgery system to capture kinematic data. In contrast, we investigate a method
for automatic, objective skill assessment that requires video data only. This
has the advantage that video can be collected effortlessly during minimally
invasive and robot-assisted training scenarios.
Methods: Our method builds on recent advances in deep learning-based video
classification. Specifically, we propose to use an inflated 3D ConvNet to
classify snippets, i.e., stacks of a few consecutive frames, extracted from
surgical video. The network is extended into a Temporal Segment Network during
training.
Results: We evaluate the method on the publicly available JIGSAWS dataset,
which consists of recordings of basic robot-assisted surgery tasks performed on
a dry lab bench-top model. Our approach achieves high skill classification
accuracies ranging from 95.1% to 100.0%.
Conclusions: Our results demonstrate the feasibility of deep learning-based
assessment of technical skill from surgical video. Notably, the 3D ConvNet is
able to learn meaningful patterns directly from the data, alleviating the need
for manual feature engineering. Further evaluation will require more annotated
data for training and testing.

本研究使用深度学习方法，基于视频数据进行自动技能评估，结果表明使用 3D ConvNet 直接从数据中学习有意义的模式可以高效评估外科技术技能，使得人工特征工程不再是必须的。