Purpose: Surgical video is an important data stream for gesture recognition.
Thus, robust visual encoders for those data-streams is similarly important.
Methods: Leveraging the Bridge-Prompt framework, we fine-tune a pre-trained
vision-text model (CLIP) for gesture recognition in surgical videos. This can
utilize extensive outside video data such as text, but also make use of label
meta-data and weakly supervised contrastive losses. Results: Our experiments
show that prompt-based video encoder outperforms standard encoders in surgical
gesture recognition tasks. Notably, it displays strong performance in zero-shot
scenarios, where gestures/tasks that were not provided during the encoder
training phase are included in the prediction phase. Additionally, we measure
the benefit of inclusion text descriptions in the feature extractor training
schema. Conclusion: Bridge-Prompt and similar pre-trained+fine-tuned video
encoder models present significant visual representation for surgical robotics,
especially in gesture recognition tasks. Given the diverse range of surgical
tasks (gestures), the ability of these models to zero-shot transfer without the
need for any task (gesture) specific retraining makes them invaluable.

利用 Bridge-Prompt 框架，我们对预训练的 vision-text 模型 (CLIP) 进行微调，以在外科手术视频中进行手势识别。实验结果显示基于提示的视频编码器在外科手势识别任务中优于标准编码器，尤其在零样本情况下表现出强大性能。这些预训练 + 微调的视频编码器模型在外科机器人学中提供了重要的视觉表示，特别适用于手势识别任务。

面向手术姿势识别的零样本基于提示的视频编码器

Zero-shot Prompt-based Video Encoder for Surgical Gesture Recognition

Recently, spatiotemporal graphs have emerged as a concise and elegant manner
of representing video clips in an object-centric fashion, and have shown to be
useful for downstream tasks such as action recognition. In this work, we
investigate the use of latent spatiotemporal graphs to represent a surgical
video in terms of the constituent anatomical structures and tools and their
evolving properties over time. To build the graphs, we first predict frame-wise
graphs using a pre-trained model, then add temporal edges between nodes based
on spatial coherence and visual and semantic similarity. Unlike previous
approaches, we incorporate long-term temporal edges in our graphs to better
model the evolution of the surgical scene and increase robustness to temporary
occlusions. We also introduce a novel graph-editing module that incorporates
prior knowledge and temporal coherence to correct errors in the graph, enabling
improved downstream task performance. Using our graph representations, we
evaluate two downstream tasks, critical view of safety prediction and surgical
phase recognition, obtaining strong results that demonstrate the quality and
flexibility of the learned representations. Code is available at
github.com/CAMMA-public/SurgLatentGraph.

利用潜在时空图对外科视频进行建模，以表示其组成的解剖结构和工具随时间的变化，通过添加长期时间边增加对手术场景演化的建模，并引入新颖的图编辑模块，评估了两项下游任务，取得了强大的结果，证明了学到的表示的质量和灵活性。

将手术视频编码为隐式时空图，用于对象与解剖驱动的推理

Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and  Anatomy-Driven Reasoning

Depth estimation in surgical video plays a crucial role in many image-guided
surgery procedures. However, it is difficult and time consuming to create depth
map ground truth datasets in surgical videos due in part to inconsistent
brightness and noise in the surgical scene. Therefore, building an accurate and
robust self-supervised depth and camera ego-motion estimation system is gaining
more attention from the computer vision community. Although several
self-supervision methods alleviate the need for ground truth depth maps and
poses, they still need known camera intrinsic parameters, which are often
missing or not recorded. Moreover, the camera intrinsic prediction methods in
existing works depend heavily on the quality of datasets. In this work, we
aimed to build a self-supervised depth and ego-motion estimation system which
can predict not only accurate depth maps and camera pose, but also camera
intrinsic parameters. We proposed a cost-volume-based supervision manner to
give the system auxiliary supervision for camera parameters prediction. The
experimental results showed that the proposed method improved the accuracy of
estimated camera parameters, ego-motion, and depth estimation.

建立了一种自监督深度和相机自运动估计系统，能够预测准确的深度图、相机姿态和相机内部参数，并通过基于成本体积的监督方式提高了估计的准确性。