This work represents a large step into modern ways of fast 3D reconstruction
based on RGB camera images. Utilizing a Microsoft HoloLens 2 as a multisensor
platform that includes an RGB camera and an inertial measurement unit for
SLAM-based camera-pose determination, we train a Neural Radiance Field (NeRF)
as a neural scene representation in real-time with the acquired data from the
HoloLens. The HoloLens is connected via Wifi to a high-performance PC that is
responsible for the training and 3D reconstruction. After the data stream ends,
the training is stopped and the 3D reconstruction is initiated, which extracts
a point cloud of the scene. With our specialized inference algorithm, five
million scene points can be extracted within 1 second. In addition, the point
cloud also includes radiometry per point. Our method of 3D reconstruction
outperforms grid point sampling with NeRFs by multiple orders of magnitude and
can be regarded as a complete real-time 3D reconstruction method in a mobile
mapping setup.

本研究利用 Microsoft HoloLens 2 作为多传感器平台，包括 RGB 相机和惯性测量单元用于 SLAM，结合 Neural Radiance Field 技术实现基于 RGB 相机图像的 3D 重建，相比于网格采样的 NeRF 算法提高了数倍的性能，是一种移动 3D 地图实时重建方法。

将 HoloLens 与 Instant-NeRFs 相结合：先进的实时 3D 移动建图

Combining HoloLens with Instant-NeRFs: Advanced Real-Time 3D Mobile  Mapping

In surgical navigation, finding correspondence between preoperative plan and
intraoperative anatomy, the so-called registration task, is imperative. One
promising approach is to intraoperatively digitize anatomy and register it with
the preoperative plan. State-of-the-art commercial navigation systems implement
such approaches for pedicle screw placement in spinal fusion surgery. Although
these systems improve surgical accuracy, they are not gold standard in clinical
practice. Besides economical reasons, this may be due to their difficult
integration into clinical workflows and unintuitive navigation feedback.
Augmented Reality has the potential to overcome these limitations.
Consequently, we propose a surgical navigation approach comprising
intraoperative surface digitization for registration and intuitive holographic
navigation for pedicle screw placement that runs entirely on the Microsoft
HoloLens. Preliminary results from phantom experiments suggest that the method
may meet clinical accuracy requirements.

本研究介绍了一种使用增强现实技术的手术导航方法，涉及手术导航、注册任务、椎弓根螺丝置入等方面，使用 Microsoft HoloLens，通过对患者体表进行数字化，利用全息投影进行直观的导航反馈，初步实验表明该方法能够满足临床精度要求。