Global visual localization in LiDAR-maps, crucial for autonomous driving
applications, remains largely unexplored due to the challenging issue of
bridging the cross-modal heterogeneity gap. Popular multi-modal learning
approach Contrastive Language-Image Pre-Training (CLIP) has popularized
contrastive symmetric loss using batch construction technique by applying it to
multi-modal domains of text and image. We apply this approach to the domains of
2D image and 3D LiDAR points on the task of cross-modal localization. Our
method is explained as follows: A batch of N (image, LiDAR) pairs is
constructed so as to predict what is the right match between N X N possible
pairings across the batch by jointly training an image encoder and LiDAR
encoder to learn a multi-modal embedding space. In this way, the cosine
similarity between N positive pairings is maximized, whereas that between the
remaining negative pairings is minimized. Finally, over the obtained similarity
scores, a symmetric cross-entropy loss is optimized. To the best of our
knowledge, this is the first work to apply batched loss approach to a
cross-modal setting of image & LiDAR data and also to show Zero-shot transfer
in a visual localization setting. We conduct extensive analyses on standard
autonomous driving datasets such as KITTI and KITTI-360 datasets. Our method
outperforms state-of-the-art recall@1 accuracy on the KITTI-360 dataset by
22.4%, using only perspective images, in contrast to the state-of-the-art
approach, which utilizes the more informative fisheye images. Additionally,
this superior performance is achieved without resorting to complex
architectures. Moreover, we demonstrate the zero-shot capabilities of our model
and we beat SOTA by 8% without even training on it. Furthermore, we establish
the first benchmark for cross-modal localization on the KITTI dataset.

利用对比映射预训练（CLIP）方法在图像与 LiDAR 点云的交叉模态本地化任务中，首次应用批处理损失方法并展示了零样本转移，并在 KITTI 数据集上取得了超过当前最先进方法 22.4% 的准确性提升，而且无需复杂的网络架构。