Understanding and anticipating intraoperative events and actions is critical
for intraoperative assistance and decision-making during minimally invasive
surgery. Automated prediction of events, actions, and the following
consequences is addressed through various computational approaches with the
objective of augmenting surgeons' perception and decision-making capabilities.
We propose a predictive neural network that is capable of understanding and
predicting critical interactive aspects of surgical workflow from
intra-abdominal video, while flexibly leveraging surgical knowledge graphs. The
approach incorporates a hypergraph-transformer (HGT) structure that encodes
expert knowledge into the network design and predicts the hidden embedding of
the graph. We verify our approach on established surgical datasets and
applications, including the detection and prediction of action triplets, and
the achievement of the Critical View of Safety (CVS). Moreover, we address
specific, safety-related tasks, such as predicting the clipping of cystic duct
or artery without prior achievement of the CVS. Our results demonstrate the
superiority of our approach compared to unstructured alternatives.

为了辅助和决策微创手术中的术中事件和行动，理解并预测术中事件和行动的机器自动化预测方法是关键。我们提出了一个预测性神经网络，能够从腹腔镜视频中理解和预测外科手术过程的关键交互方面，并灵活利用外科知识图谱。该方法采用了一个超图转换器结构来将专家知识编码到网络设计中，并预测图的隐藏嵌入。我们在已有的外科数据集和应用中验证了我们的方法，包括动作三元组的检测和预测，以及安全关键视野的实现。此外，我们还解决了特定的与安全相关的任务，如在实现安全关键视野之前预测囊管或动脉的夹闭操作。我们的结果表明，相比于非结构化的替代方法，我们的方法具有优势。

超图 - 变换器（HGT）用于腹腔镜和机器人手术中的交互式事件预测

Hypergraph-Transformer (HGT) for Interactive Event Prediction in  Laparoscopic and Robotic Surgery

While great strides have been made in using deep learning algorithms to solve
supervised learning tasks, the problem of unsupervised learning - leveraging
unlabeled examples to learn about the structure of a domain - remains a
difficult unsolved challenge. Here, we explore prediction of future frames in a
video sequence as an unsupervised learning rule for learning about the
structure of the visual world. We describe a predictive neural network
("PredNet") architecture that is inspired by the concept of "predictive coding"
from the neuroscience literature. These networks learn to predict future frames
in a video sequence, with each layer in the network making local predictions
and only forwarding deviations from those predictions to subsequent network
layers. We show that these networks are able to robustly learn to predict the
movement of synthetic (rendered) objects, and that in doing so, the networks
learn internal representations that are useful for decoding latent object
parameters (e.g. pose) that support object recognition with fewer training
views. We also show that these networks can scale to complex natural image
streams (car-mounted camera videos), capturing key aspects of both egocentric
movement and the movement of objects in the visual scene, and the
representation learned in this setting is useful for estimating the steering
angle. Altogether, these results suggest that prediction represents a powerful
framework for unsupervised learning, allowing for implicit learning of object
and scene structure.

通过建立一个具有预测编码特点的预测神经网络 (PredNet)，本文探讨了将预测未来的视频帧作为无监督学习的规则来学习视觉世界结构的问题。 在对计算机生成的物体的运动进行鲁棒的学习的同时，也能够缩短学习视野并从中获得内部表示。此外，这种网络架构也能够将检测到的流和视觉场景中的物体运动进行深度学习，为估算自我位置提供支持。 综上所述，预测是无监督学习的强大框架之一，能够进行隐式的物体和场景结构学习。