Surgery monitoring in Mixed Reality (MR) environments has recently received substantial focus due to its importance in image-based decisions, skill assessment, and robot-assisted surgery. Tracking hands and articulated surgical instruments is crucial for the success of these applications. Due to the lack of annotated datasets and the complexity of the task, only a few works have addressed this problem. In this work, we present SurgeoNet, a real-time neural network pipeline to accurately detect and track surgical instruments from a stereo VR view. Our multi-stage approach is inspired by state-of-the-art neural-network architectural design, like YOLO and Transformers. We demonstrate the generalization capabilities of SurgeoNet in challenging real-world scenarios, achieved solely through training on synthetic data. The approach can be easily extended to any new set of articulated surgical instruments. SurgeoNet's code and data are publicly available.

本研究解决了混合现实环境下手术监控中缺乏标注数据和任务复杂性的问题，通过提出SurgeoNet，一个实时神经网络管道，能够准确检测和跟踪手术器械。该方法展示了在仅用合成数据训练时SurgeoNet在复杂现实场景中的泛化能力，具有显著的实际应用潜力。

SurgeoNet：实时三维姿态估计手术器械的合成训练网络