We propose a novel visual re-localization method based on direct matching
between the implicit 3D descriptors and the 2D image with transformer. A
conditional neural radiance field(NeRF) is chosen as the 3D scene
representation in our pipeline, which supports continuous 3D descriptors
generation and neural rendering. By unifying the feature matching and the scene
coordinate regression to the same framework, our model learns both
generalizable knowledge and scene prior respectively during two training
stages. Furthermore, to improve the localization robustness when domain gap
exists between training and testing phases, we propose an appearance adaptation
layer to explicitly align styles between the 3D model and the query image.
Experiments show that our method achieves higher localization accuracy than
other learning-based approaches on multiple benchmarks. Code is available at
https://github.com/JenningsL/nerf-loc.

本文提出了一种新颖的基于 3D 场景描述符和转换器与 2D 图像进行直接匹配的视觉重新定位方法，NeRF 作为 3D 场景表示，在统一特征匹配和场景坐标回归的同时，本模型在两个训练阶段学习可推广的知识和场景先验知识。此外，为了提高本模型在训练和测试间的鲁棒性，本文提出了一个外观适应层来明确地对齐 3D 模型和查询图像。实验证明，本方法在多个基准测试上实现了比其他基于学习的方法更高的定位精度。

使用有条件的神经辐射场进行视觉定位的 NeRF-Loc

NeRF-Loc: Visual Localization with Conditional Neural Radiance Field

Visual re-localization aims to recover camera poses in a known environment,
which is vital for applications like robotics or augmented reality.
Feed-forward absolute camera pose regression methods directly output poses by a
network, but suffer from low accuracy. Meanwhile, scene coordinate based
methods are accurate, but need iterative RANSAC post-processing, which brings
challenges to efficient end-to-end training and inference. In order to have the
best of both worlds, we propose a feed-forward method termed SC-wLS that
exploits all scene coordinate estimates for weighted least squares pose
regression. This differentiable formulation exploits a weight network imposed
on 2D-3D correspondences, and requires pose supervision only. Qualitative
results demonstrate the interpretability of learned weights. Evaluations on
7Scenes and Cambridge datasets show significantly promoted performance when
compared with former feed-forward counterparts. Moreover, our SC-wLS method
enables a new capability: self-supervised test-time adaptation on the weight
network. Codes and models are publicly available.

本文提出了一种名为 SC-wLS 的前馈方法，该方法利用所有场景坐标估计来进行加权最小二乘姿态回归，通过被施加在 2D-3D 对应关系上的权重网络进行可区分的公式化，且只需要姿态监督，通过 7scenes 和剑桥数据集的评估表明，与之前的前馈方法相比，SC-wLS 方法显著提高了性能。此外，我们的 SC-wLS 方法还实现了一种新的功能：加权网络的自我监督测试时间适应。

SC-wLS：面向可解释的前馈相机重定位

SC-wLS: Towards Interpretable Feed-forward Camera Re-localization

Visual re-localization means using a single image as input to estimate the
camera's location and orientation relative to a pre-recorded environment. The
highest-scoring methods are "structure based," and need the query camera's
intrinsics as an input to the model, with careful geometric optimization. When
intrinsics are absent, methods vie for accuracy by making various other
assumptions. This yields fairly good localization scores, but the models are
"narrow" in some way, eg., requiring costly test-time computations, or depth
sensors, or multiple query frames. In contrast, our proposed method makes few
special assumptions, and is fairly lightweight in training and testing.
Our pose regression network learns from only relative poses of training
scenes. For inference, it builds a graph connecting the query image to training
counterparts and uses a graph neural network (GNN) with image representations
on nodes and image-pair representations on edges. By efficiently passing
messages between them, both representation types are refined to produce a
consistent camera pose estimate. We validate the effectiveness of our approach
on both standard indoor (7-Scenes) and outdoor (Cambridge Landmarks) camera
re-localization benchmarks. Our relative pose regression method matches the
accuracy of absolute pose regression networks, while retaining the
relative-pose models' test-time speed and ability to generalize to non-training
scenes.

本研究提出了一种基于姿态回归网络和图神经网络相结合的视觉重新定位方法，可在没有内参信息的情况下达到与绝对姿态回归网络相当的准确性，同时保持相对姿态回归模型的高测试速度和对非训练场景的适用性。