Purpose: Surgical video is an important data stream for gesture recognition.
Thus, robust visual encoders for those data-streams is similarly important.
Methods: Leveraging the Bridge-Prompt framework, we fine-tune a pre-trained
vision-text model (CLIP) for gesture recognition in surgical videos. This can
utilize extensive outside video data such as text, but also make use of label
meta-data and weakly supervised contrastive losses. Results: Our experiments
show that prompt-based video encoder outperforms standard encoders in surgical
gesture recognition tasks. Notably, it displays strong performance in zero-shot
scenarios, where gestures/tasks that were not provided during the encoder
training phase are included in the prediction phase. Additionally, we measure
the benefit of inclusion text descriptions in the feature extractor training
schema. Conclusion: Bridge-Prompt and similar pre-trained+fine-tuned video
encoder models present significant visual representation for surgical robotics,
especially in gesture recognition tasks. Given the diverse range of surgical
tasks (gestures), the ability of these models to zero-shot transfer without the
need for any task (gesture) specific retraining makes them invaluable.

利用 Bridge-Prompt 框架，我们对预训练的 vision-text 模型 (CLIP) 进行微调，以在外科手术视频中进行手势识别。实验结果显示基于提示的视频编码器在外科手势识别任务中优于标准编码器，尤其在零样本情况下表现出强大性能。这些预训练 + 微调的视频编码器模型在外科机器人学中提供了重要的视觉表示，特别适用于手势识别任务。