Natural language could play an important role in developing generalist
surgical models by providing a broad source of supervision from raw texts. This
flexible form of supervision can enable the model's transferability across
datasets and tasks as natural language can be used to reference learned visual
concepts or describe new ones. In this work, we present HecVL, a novel
hierarchical video-language pretraining approach for building a generalist
surgical model. Specifically, we construct a hierarchical video-text paired
dataset by pairing the surgical lecture video with three hierarchical levels of
texts: at clip-level, atomic actions using transcribed audio texts; at
phase-level, conceptual text summaries; and at video-level, overall abstract
text of the surgical procedure. Then, we propose a novel fine-to-coarse
contrastive learning framework that learns separate embedding spaces for the
three video-text hierarchies using a single model. By disentangling embedding
spaces of different hierarchical levels, the learned multi-modal
representations encode short-term and long-term surgical concepts in the same
model. Thanks to the injected textual semantics, we demonstrate that the HecVL
approach can enable zero-shot surgical phase recognition without any human
annotation. Furthermore, we show that the same HecVL model for surgical phase
recognition can be transferred across different surgical procedures and medical
centers.

通过使用自然语言实现手术模型的普适性训练，本研究提出了一种名为 HecVL 的新型分层视频 - 文本预训练方法，其中通过构建层次化的视频 - 文本配对数据集，通过剪辑级、阶段级和视频级的文本信息学习了多模态嵌入空间，并使用对比学习的框架进行训练，使模型能够实现零样本手术阶段识别，并且在不同手术程序和医疗中心之间实现了模型的迁移。