Global visual localization in LiDAR-maps, crucial for autonomous driving
applications, remains largely unexplored due to the challenging issue of
bridging the cross-modal heterogeneity gap. Popular multi-modal learning
approach Contrastive Language-Image Pre-Training (CLIP) has popularized
contrastive symmetric loss using batch construction technique by applying it to
multi-modal domains of text and image. We apply this approach to the domains of
2D image and 3D LiDAR points on the task of cross-modal localization. Our
method is explained as follows: A batch of N (image, LiDAR) pairs is
constructed so as to predict what is the right match between N X N possible
pairings across the batch by jointly training an image encoder and LiDAR
encoder to learn a multi-modal embedding space. In this way, the cosine
similarity between N positive pairings is maximized, whereas that between the
remaining negative pairings is minimized. Finally, over the obtained similarity
scores, a symmetric cross-entropy loss is optimized. To the best of our
knowledge, this is the first work to apply batched loss approach to a
cross-modal setting of image & LiDAR data and also to show Zero-shot transfer
in a visual localization setting. We conduct extensive analyses on standard
autonomous driving datasets such as KITTI and KITTI-360 datasets. Our method
outperforms state-of-the-art recall@1 accuracy on the KITTI-360 dataset by
22.4%, using only perspective images, in contrast to the state-of-the-art
approach, which utilizes the more informative fisheye images. Additionally,
this superior performance is achieved without resorting to complex
architectures. Moreover, we demonstrate the zero-shot capabilities of our model
and we beat SOTA by 8% without even training on it. Furthermore, we establish
the first benchmark for cross-modal localization on the KITTI dataset.

利用对比映射预训练（CLIP）方法在图像与 LiDAR 点云的交叉模态本地化任务中，首次应用批处理损失方法并展示了零样本转移，并在 KITTI 数据集上取得了超过当前最先进方法 22.4% 的准确性提升，而且无需复杂的网络架构。

LIP-Loc: 跨模态定位的激光雷达图像预训练

LIP-Loc: LiDAR Image Pretraining for Cross-Modal Localization

Global visual localization estimates the absolute pose of a camera using a
single image, in a previously mapped area. Obtaining the pose from a single
image enables many robotics and augmented/virtual reality applications.
Inspired by latest advances in deep learning, many existing approaches directly
learn and regress 6 DoF pose from an input image. However, these methods do not
fully utilize the underlying scene geometry for pose regression. The challenge
in monocular relocalization is the minimal availability of supervised training
data, which is just the corresponding 6 DoF poses of the images. In this paper,
we propose to utilize these minimal available labels (.i.e, poses) to learn the
underlying 3D geometry of the scene and use the geometry to estimate the 6 DoF
camera pose. We present a learning method that uses these pose labels and rigid
alignment to learn two 3D geometric representations (\textit{X, Y, Z
coordinates}) of the scene, one in camera coordinate frame and the other in
global coordinate frame. Given a single image, it estimates these two 3D scene
representations, which are then aligned to estimate a pose that matches the
pose label. This formulation allows for the active inclusion of additional
learning constraints to minimize 3D alignment errors between the two 3D scene
representations, and 2D re-projection errors between the 3D global scene
representation and 2D image pixels, resulting in improved localization
accuracy. During inference, our model estimates the 3D scene geometry in camera
and global frames and aligns them rigidly to obtain pose in real-time. We
evaluate our work on three common visual localization datasets, conduct
ablation studies, and show that our method exceeds state-of-the-art regression
methods' pose accuracy on all datasets.

通过利用少量的标签数据（即相机姿态），本文提出了一种学习方法，将这些标签与刚性对齐相结合，学习场景的三维几何表示，并将其用于估计相机的六自由度姿态，从而提高了定位的准确性。