We introduce a novel problem, i.e., the localization of an input image within
a multi-modal reference map represented by a database of 3D scene graphs. These
graphs comprise multiple modalities, including object-level point clouds,
images, attributes, and relationships between objects, offering a lightweight
and efficient alternative to conventional methods that rely on extensive image
databases. Given the available modalities, the proposed method SceneGraphLoc
learns a fixed-sized embedding for each node (i.e., representing an object
instance) in the scene graph, enabling effective matching with the objects
visible in the input query image. This strategy significantly outperforms other
cross-modal methods, even without incorporating images into the map embeddings.
When images are leveraged, SceneGraphLoc achieves performance close to that of
state-of-the-art techniques depending on large image databases, while requiring
three orders-of-magnitude less storage and operating orders-of-magnitude
faster. The code will be made public.

这篇论文介绍了一个新问题，即在由 3D 场景图数据库表示的多模式参考地图中定位输入图像。通过学习场景图中每个节点的固定大小嵌入，该方法提供了一种轻量级且高效的替代方法，以有效地匹配输入查询图像中可见的对象实例，实现了超出其他跨模态方法的性能表现，并在使用图像时实现了与大型图像数据库依赖的最先进技术相当的性能，同时存储效率提高了三个数量级并且运行速度加快了几个数量级。

SceneGraphLoc: 跨模态粗略基于 3D 场景图的视觉定位

SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphs

Place recognition is an important task for robots and autonomous cars to
localize themselves and close loops in pre-built maps. While single-modal
sensor-based methods have shown satisfactory performance, cross-modal place
recognition that retrieving images from a point-cloud database remains a
challenging problem. Current cross-modal methods transform images into 3D
points using depth estimation for modality conversion, which are usually
computationally intensive and need expensive labeled data for depth
supervision. In this work, we introduce a fast and lightweight framework to
encode images and point clouds into place-distinctive descriptors. We propose
an effective Field of View (FoV) transformation module to convert point clouds
into an analogous modality as images. This module eliminates the necessity for
depth estimation and helps subsequent modules achieve real-time performance. We
further design a non-negative factorization-based encoder to extract mutually
consistent semantic features between point clouds and images. This encoder
yields more distinctive global descriptors for retrieval. Experimental results
on the KITTI dataset show that our proposed methods achieve state-of-the-art
performance while running in real time. Additional evaluation on the HAOMO
dataset covering a 17 km trajectory further shows the practical generalization
capabilities. We have released the implementation of our methods as open source
at: this https URL

通过引入快速轻量级的框架，将图像和点云转换成具有地点鉴别性的描述符。实验结果表明，所提出的方法在实时执行的同时达到了最先进的性能，具有实际的泛化能力。