Understanding and anticipating intraoperative events and actions is critical
for intraoperative assistance and decision-making during minimally invasive
surgery. Automated prediction of events, actions, and the following
consequences is addressed through various computational approaches with the
objective of augmenting surgeons' perception and decision-making capabilities.
We propose a predictive neural network that is capable of understanding and
predicting critical interactive aspects of surgical workflow from
intra-abdominal video, while flexibly leveraging surgical knowledge graphs. The
approach incorporates a hypergraph-transformer (HGT) structure that encodes
expert knowledge into the network design and predicts the hidden embedding of
the graph. We verify our approach on established surgical datasets and
applications, including the detection and prediction of action triplets, and
the achievement of the Critical View of Safety (CVS). Moreover, we address
specific, safety-related tasks, such as predicting the clipping of cystic duct
or artery without prior achievement of the CVS. Our results demonstrate the
superiority of our approach compared to unstructured alternatives.

为了辅助和决策微创手术中的术中事件和行动，理解并预测术中事件和行动的机器自动化预测方法是关键。我们提出了一个预测性神经网络，能够从腹腔镜视频中理解和预测外科手术过程的关键交互方面，并灵活利用外科知识图谱。该方法采用了一个超图转换器结构来将专家知识编码到网络设计中，并预测图的隐藏嵌入。我们在已有的外科数据集和应用中验证了我们的方法，包括动作三元组的检测和预测，以及安全关键视野的实现。此外，我们还解决了特定的与安全相关的任务，如在实现安全关键视野之前预测囊管或动脉的夹闭操作。我们的结果表明，相比于非结构化的替代方法，我们的方法具有优势。