Environment maps endowed with sophisticated semantics are pivotal for
facilitating seamless interaction between robots and humans, enabling them to
effectively carry out various tasks. Open-vocabulary maps, powered by
Visual-Language models (VLMs), possess inherent advantages, including
multimodal retrieval and open-set classes. However, existing open-vocabulary
maps are constrained to closed indoor scenarios and VLM features, thereby
diminishing their usability and inference capabilities. Moreover, the absence
of topological relationships further complicates the accurate querying of
specific instances. In this work, we propose OpenGraph, a representation of
open-vocabulary hierarchical graph structure designed for large-scale outdoor
environments. OpenGraph initially extracts instances and their captions from
visual images using 2D foundation models, encoding the captions with features
to enhance textual reasoning. Subsequently, 3D incremental panoramic mapping
with feature embedding is achieved by projecting images onto LiDAR point
clouds. Finally, the environment is segmented based on lane graph connectivity
to construct a hierarchical graph. Validation results from real public dataset
SemanticKITTI demonstrate that, even without fine-tuning the models, OpenGraph
exhibits the ability to generalize to novel semantic classes and achieve the
highest segmentation and query accuracy. The source code of OpenGraph is
publicly available at this https URL

本文提出了 OpenGraph，一种适用于大规模户外环境的开放词汇层次图结构表示，该结构兼具图像和文字语义，并通过 2D 模型提取实例和标题，以增强文本推理能力，并通过将图像投影到 LiDAR 点云上实现了 3D 增量全景映射和特征嵌入，最终根据车道图连通性对环境进行分割构建了一个层次化图。验证结果显示，OpenGraph 能够在没有微调模型的情况下推广到新的语义类别，并取得最高的分割和查询精度。

OpenGraph: 大规模户外环境中的开放词汇层次化三维图谱表示

OpenGraph: Open-Vocabulary Hierarchical 3D Graph Representation in  Large-Scale Outdoor Environments

Rapid and reliable identification of dynamic scene parts, also known as
motion segmentation, is a key challenge for mobile sensors. Contemporary RGB
camera-based methods rely on modeling camera and scene properties however, are
often under-constrained and fall short in unknown categories. Event cameras
have the potential to overcome these limitations, but corresponding methods
have only been demonstrated in smaller-scale indoor environments with
simplified dynamic objects. This work presents an event-based method for
class-agnostic motion segmentation that can successfully be deployed across
complex large-scale outdoor environments too. To this end, we introduce a novel
divide-and-conquer pipeline that combines: (a) ego-motion compensated events,
computed via a scene understanding module that predicts monocular depth and
camera pose as auxiliary tasks, and (b) optical flow from a dedicated optical
flow module. These intermediate representations are then fed into a
segmentation module that predicts motion segmentation masks. A novel
transformer-based temporal attention module in the segmentation module builds
correlations across adjacent 'frames' to get temporally consistent segmentation
masks. Our method sets the new state-of-the-art on the classic EV-IMO benchmark
(indoors), where we achieve improvements of 2.19 moving object IoU (2.22 mIoU)
and 4.52 point IoU respectively, as well as on a newly-generated motion
segmentation and tracking benchmark (outdoors) based on the DSEC event dataset,
termed DSEC-MOTS, where we show improvement of 12.91 moving object IoU.

本文介绍了一种基于事件相机的运动分割方法，可以成功应用于复杂的大规模室外环境，采用创新的分而治之流程结合状态理解模块和光流模块，以及基于 Transformer 的时间关注模块进行运动分割，在经典的室内 EV-IMO 基准测试和基于 DSEC 事件数据集的新生成的动态分割和跟踪基准测试中取得了最新的最佳表现。