Open-vocabulary 3D Object Detection (OV-3DDet) addresses the detection of
objects from an arbitrary list of novel categories in 3D scenes, which remains
a very challenging problem. In this work, we propose CoDAv2, a unified
framework designed to innovatively tackle both the localization and
classification of novel 3D objects, under the condition of limited base
categories. For localization, the proposed 3D Novel Object Discovery (3D-NOD)
strategy utilizes 3D geometries and 2D open-vocabulary semantic priors to
discover pseudo labels for novel objects during training. 3D-NOD is further
extended with an Enrichment strategy that significantly enriches the novel
object distribution in the training scenes, and then enhances the model's
ability to localize more novel objects. The 3D-NOD with Enrichment is termed
3D-NODE. For classification, the Discovery-driven Cross-modal Alignment (DCMA)
module aligns features from 3D point clouds and 2D/textual modalities,
employing both class-agnostic and class-specific alignments that are
iteratively refined to handle the expanding vocabulary of objects. Besides, 2D
box guidance boosts the classification accuracy against complex background
noises, which is coined as Box-DCMA. Extensive evaluation demonstrates the
superiority of CoDAv2. CoDAv2 outperforms the best-performing method by a large
margin (AP_Novel of 9.17 vs. 3.61 on SUN-RGBD and 9.12 vs. 3.74 on ScanNetv2).
Source code and pre-trained models are available at the GitHub project page.

CoDAv2 是一个统一的框架，用于创新性地解决有限基本类别条件下的新型 3D 物体的定位和分类问题，其中使用 3D 几何和 2D 开放词汇语义先验来发现训练期间新型物体的伪标签以及通过 3D 点云和 2D / 文本模态之间的特征对齐来提升分类准确性。

合作式新颖物体发现与基于盒子引导的跨模态对齐在开放词汇 3D 物体检测中的应用

Collaborative Novel Object Discovery and Box-Guided Cross-Modal  Alignment for Open-Vocabulary 3D Object Detection

Open-vocabulary 3D Object Detection (OV-3DDet) aims to detect objects from an
arbitrary list of categories within a 3D scene, which remains seldom explored
in the literature. There are primarily two fundamental problems in OV-3DDet,
i.e., localizing and classifying novel objects. This paper aims at addressing
the two problems simultaneously via a unified framework, under the condition of
limited base categories. To localize novel 3D objects, we propose an effective
3D Novel Object Discovery strategy, which utilizes both the 3D box geometry
priors and 2D semantic open-vocabulary priors to generate pseudo box labels of
the novel objects. To classify novel object boxes, we further develop a
cross-modal alignment module based on discovered novel boxes, to align feature
spaces between 3D point cloud and image/text modalities. Specifically, the
alignment process contains a class-agnostic and a class-discriminative
alignment, incorporating not only the base objects with annotations but also
the increasingly discovered novel objects, resulting in an iteratively enhanced
alignment. The novel box discovery and crossmodal alignment are jointly learned
to collaboratively benefit each other. The novel object discovery can directly
impact the cross-modal alignment, while a better feature alignment can, in
turn, boost the localization capability, leading to a unified OV-3DDet
framework, named CoDA, for simultaneous novel object localization and
classification. Extensive experiments on two challenging datasets (i.e.,
SUN-RGBD and ScanNet) demonstrate the effectiveness of our method and also show
a significant mAP improvement upon the best-performing alternative method by
80%. Codes and pre-trained models are released on the project page.

通过统一的框架，在有限的基础类别的条件下，本论文旨在同时解决定位和分类新颖目标的两个问题，提出了一种有效的 3D 新目标发现策略以及基于发现的新目标的跨模态对齐模块，通过迭代增强的对齐来统一实现了开放词汇 3D 目标检测的既定框架。

CoDA: 协作式创新箱发现与开放式词汇 3D 物体检测的交叉模态对齐

CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for  Open-vocabulary 3D Object Detection

Point cloud-based open-vocabulary 3D object detection aims to detect 3D
categories that do not have ground-truth annotations in the training set. It is
extremely challenging because of the limited data and annotations (bounding
boxes with class labels or text descriptions) of 3D scenes. Previous approaches
leverage large-scale richly-annotated image datasets as a bridge between 3D and
category semantics but require an extra alignment process between 2D images and
3D points, limiting the open-vocabulary ability of 3D detectors. Instead of
leveraging 2D images, we propose Object2Scene, the first approach that
leverages large-scale large-vocabulary 3D object datasets to augment existing
3D scene datasets for open-vocabulary 3D object detection. Object2Scene inserts
objects from different sources into 3D scenes to enrich the vocabulary of 3D
scene datasets and generates text descriptions for the newly inserted objects.
We further introduce a framework that unifies 3D detection and visual
grounding, named L3Det, and propose a cross-domain category-level contrastive
learning approach to mitigate the domain gap between 3D objects from different
datasets. Extensive experiments on existing open-vocabulary 3D object detection
benchmarks show that Object2Scene obtains superior performance over existing
methods. We further verify the effectiveness of Object2Scene on a new benchmark
OV-ScanNet-200, by holding out all rare categories as novel categories not seen
during training.

Object2Scene 和 L3Det 是用于开放词汇 3D 物体检测的首个方法，通过引入大规模的 3D 物体数据集，将物体插入到 3D 场景中来丰富 3D 场景数据集的词汇，并在不同数据集的 3D 对象之间缓解领域差异，并证实了其在 OV-ScanNet-200 基准测试中的有效性。