Autonomous robot systems have attracted increasing research attention in
recent years, where environment understanding is a crucial step for robot
navigation, human-robot interaction, and decision. Real-world robot systems
usually collect visual data from multiple sensors and are required to recognize
numerous objects and their movements in complex human-crowded settings.
Traditional benchmarks, with their reliance on single sensors and limited
object classes and scenarios, fail to provide the comprehensive environmental
understanding robots need for accurate navigation, interaction, and
decision-making. As an extension of JRDB dataset, we unveil JRDB-PanoTrack, a
novel open-world panoptic segmentation and tracking benchmark, towards more
comprehensive environmental perception. JRDB-PanoTrack includes (1) various
data involving indoor and outdoor crowded scenes, as well as comprehensive 2D
and 3D synchronized data modalities; (2) high-quality 2D spatial panoptic
segmentation and temporal tracking annotations, with additional 3D label
projections for further spatial understanding; (3) diverse object classes for
closed- and open-world recognition benchmarks, with OSPA-based metrics for
evaluation. Extensive evaluation of leading methods shows significant
challenges posed by our dataset.

自主机器人系统研究中，环境理解是机器人导航、人机交互和决策的关键步骤。本文介绍了一种新的开放世界全景分割和跟踪基准数据集 JRDB-PanoTrack，旨在提供更全面的环境感知。该数据集包含室内和室外拥挤场景的多种数据、包括 2D 和 3D 同步数据模态；高质量的 2D 空间全景分割和时间跟踪标注，并附加 3D 标签投影以进一步提供空间理解；适用于封闭世界和开放世界识别基准的各种对象类别，使用基于 OSPA 的度量进行评估。对主要方法的广泛评估表明，我们的数据集提出了重要挑战。