We propose a novel visual re-localization method based on direct matching
between the implicit 3D descriptors and the 2D image with transformer. A
conditional neural radiance field(NeRF) is chosen as the 3D scene
representation in our pipeline, which supports continuous 3D descriptors
generation and neural rendering. By unifying the feature matching and the scene
coordinate regression to the same framework, our model learns both
generalizable knowledge and scene prior respectively during two training
stages. Furthermore, to improve the localization robustness when domain gap
exists between training and testing phases, we propose an appearance adaptation
layer to explicitly align styles between the 3D model and the query image.
Experiments show that our method achieves higher localization accuracy than
other learning-based approaches on multiple benchmarks. Code is available at
https://github.com/JenningsL/nerf-loc.

本文提出了一种新颖的基于 3D 场景描述符和转换器与 2D 图像进行直接匹配的视觉重新定位方法，NeRF 作为 3D 场景表示，在统一特征匹配和场景坐标回归的同时，本模型在两个训练阶段学习可推广的知识和场景先验知识。此外，为了提高本模型在训练和测试间的鲁棒性，本文提出了一个外观适应层来明确地对齐 3D 模型和查询图像。实验证明，本方法在多个基准测试上实现了比其他基于学习的方法更高的定位精度。