This paper presents a framework that combines traditional keypoint-based camera pose optimization with an invertible neural rendering mechanism. Our proposed 3D scene representation, Nerfels, is locally dense yet globally sparse. As opposed to existing invertible neural rendering systems which overfit a model to the entire scene, we adopt a feature-driven approach for representing scene-agnostic, local 3D patches with renderable codes. By modelling a scene only where local features are detected, our framework effectively generalizes to unseen local regions in the scene via an optimizable code conditioning mechanism in the neural renderer, all while maintaining the low memory footprint of a sparse 3D map representation. Our model can be incorporated to existing state-of-the-art hand-crafted and learned local feature pose estimators, yielding improved performance when evaluating on ScanNet for wide camera baseline scenarios.

本文提出了一个框架，它将传统的基于关键点的相机姿态优化与可逆神经渲染机制相结合。我们提出的3D场景表示Nerfels在本地密集而全局稀疏。通过仅在检测到本地特征的位置建模场景，我们的框架通过神经渲染器中的可优化代码调节机制有效地概括了场景中的未见局部区域，同时保持了稀疏3D地图表示的低内存占用。该模型可与现有的最先进的手工制作和学习本地特征姿态估计器相结合，提高在ScanNet宽摄像机基线场景评估时的性能。

Nerfels: 基于可渲染神经码的相机姿态估计改进