This study tackles key obstacles in adopting surgical navigation in
orthopedic surgeries, including time, cost, radiation, and workflow integration
challenges. Recently, our work X23D showed an approach for generating 3D
anatomical models of the spine from only a few intraoperative fluoroscopic
images. This negates the need for conventional registration-based surgical
navigation by creating a direct intraoperative 3D reconstruction of the
anatomy. Despite these strides, the practical application of X23D has been
limited by a domain gap between synthetic training data and real intraoperative
images.
In response, we devised a novel data collection protocol for a paired dataset
consisting of synthetic and real fluoroscopic images from the same
perspectives. Utilizing this dataset, we refined our deep learning model via
transfer learning, effectively bridging the domain gap between synthetic and
real X-ray data. A novel style transfer mechanism also allows us to convert
real X-rays to mirror the synthetic domain, enabling our in-silico-trained X23D
model to achieve high accuracy in real-world settings.
Our results demonstrated that the refined model can rapidly generate accurate
3D reconstructions of the entire lumbar spine from as few as three
intraoperative fluoroscopic shots. It achieved an 84% F1 score, matching the
accuracy of our previous synthetic data-based research. Additionally, with a
computational time of only 81.1 ms, our approach provides real-time
capabilities essential for surgery integration.
Through examining ideal imaging setups and view angle dependencies, we've
further confirmed our system's practicality and dependability in clinical
settings. Our research marks a significant step forward in intraoperative 3D
reconstruction, offering enhancements to surgical planning, navigation, and
robotics.

我们的研究通过解决手术导航中的时间、成本、辐射和工作流整合等关键障碍，提出了一个从少量术中透视 X 光图像生成脊柱三维解剖模型的方法，并通过收集体现真实和虚拟透视 X 光图像的配对数据集，利用迁移学习和风格转移机制提高了深度学习模型在真实术中环境中的准确性，研究成果表明我们的方法在手术规划、导航和机器人技术领域有重要的应用潜力。

基于真实荧光透视数据的腰椎三维重建领域适应策略

Domain adaptation strategies for 3D reconstruction of the lumbar spine  using real fluoroscopy data

The accurate reconstruction of surgical scenes from surgical videos is
critical for various applications, including intraoperative navigation and
image-guided robotic surgery automation. However, previous approaches, mainly
relying on depth estimation, have limited effectiveness in reconstructing
surgical scenes with moving surgical tools. To address this limitation and
provide accurate 3D position prediction for surgical tools in all frames, we
propose a novel approach called SAMSNeRF that combines Segment Anything Model
(SAM) and Neural Radiance Field (NeRF) techniques. Our approach generates
accurate segmentation masks of surgical tools using SAM, which guides the
refinement of the dynamic surgical scene reconstruction by NeRF. Our
experimental results on public endoscopy surgical videos demonstrate that our
approach successfully reconstructs high-fidelity dynamic surgical scenes and
accurately reflects the spatial information of surgical tools. Our proposed
approach can significantly enhance surgical navigation and automation by
providing surgeons with accurate 3D position information of surgical tools
during surgery.The source code will be released soon.

我们提出了一种名为 SAMSNeRF 的创新方法，结合了 Segment Anything Model（SAM）和 Neural Radiance Field（NeRF）技术，通过 SAM 生成手术工具的准确分割掩模，在 NeRF 的指导下对动态手术场景进行精细重建，实验结果表明我们的方法成功重建了具有高保真度的动态手术场景，并准确反映了手术工具的空间信息，从而大大提升了手术导航和自动化的效果。

SAMSNeRF: 通过神经辐射场（NeRF）的动态手术场景重建，由 Segment Anything Model (SAM) 指导

SAMSNeRF: Segment Anything Model (SAM) Guides Dynamic Surgical Scene  Reconstruction by Neural Radiance Field (NeRF)

In surgical navigation, finding correspondence between preoperative plan and
intraoperative anatomy, the so-called registration task, is imperative. One
promising approach is to intraoperatively digitize anatomy and register it with
the preoperative plan. State-of-the-art commercial navigation systems implement
such approaches for pedicle screw placement in spinal fusion surgery. Although
these systems improve surgical accuracy, they are not gold standard in clinical
practice. Besides economical reasons, this may be due to their difficult
integration into clinical workflows and unintuitive navigation feedback.
Augmented Reality has the potential to overcome these limitations.
Consequently, we propose a surgical navigation approach comprising
intraoperative surface digitization for registration and intuitive holographic
navigation for pedicle screw placement that runs entirely on the Microsoft
HoloLens. Preliminary results from phantom experiments suggest that the method
may meet clinical accuracy requirements.

本研究介绍了一种使用增强现实技术的手术导航方法，涉及手术导航、注册任务、椎弓根螺丝置入等方面，使用 Microsoft HoloLens，通过对患者体表进行数字化，利用全息投影进行直观的导航反馈，初步实验表明该方法能够满足临床精度要求。