Closed-set 3D perception models trained on only a pre-defined set of object
categories can be inadequate for safety critical applications such as
autonomous driving where new object types can be encountered after deployment.
In this paper, we present a multi-modal auto labeling pipeline capable of
generating amodal 3D bounding boxes and tracklets for training models on
open-set categories without 3D human labels. Our pipeline exploits motion cues
inherent in point cloud sequences in combination with the freely available 2D
image-text pairs to identify and track all traffic participants. Compared to
the recent studies in this domain, which can only provide class-agnostic auto
labels limited to moving objects, our method can handle both static and moving
objects in the unsupervised manner and is able to output open-vocabulary
semantic labels thanks to the proposed vision-language knowledge distillation.
Experiments on the Waymo Open Dataset show that our approach outperforms the
prior work by significant margins on various unsupervised 3D perception tasks.

本文介绍了一种多模态自动标注流程，可以生成用于训练开放式类别的无标注 3D 边界框和轨迹，从而处理自动驾驶等安全关键应用中可能在部署后遇到的新物体类型。相比于当前领域的最新研究，我们的方法可以以无监督的方式处理静态和移动的对象，并通过提出的视觉 - 语言知识蒸馏方法输出开放式词汇的语义标签。基于 Waymo 开放数据集的实验证明，我们的方法在各种无监督 3D 感知任务上显著优于先前的工作。