In this work, we tackle the limitations of current LiDAR-based 3D object
detection systems, which are hindered by a restricted class vocabulary and the
high costs associated with annotating new object classes. Our exploration of
open-vocabulary (OV) learning in urban environments aims to capture novel
instances using pre-trained vision-language models (VLMs) with multi-sensor
data. We design and benchmark a set of four potential solutions as baselines,
categorizing them into either top-down or bottom-up approaches based on their
input data strategies. While effective, these methods exhibit certain
limitations, such as missing novel objects in 3D box estimation or applying
rigorous priors, leading to biases towards objects near the camera or of
rectangular geometries. To overcome these limitations, we introduce a universal
\textsc{Find n' Propagate} approach for 3D OV tasks, aimed at maximizing the
recall of novel objects and propagating this detection capability to more
distant areas thereby progressively capturing more. In particular, we utilize a
greedy box seeker to search against 3D novel boxes of varying orientations and
depth in each generated frustum and ensure the reliability of newly identified
boxes by cross alignment and density ranker. Additionally, the inherent bias
towards camera-proximal objects is alleviated by the proposed remote simulator,
which randomly diversifies pseudo-labeled novel instances in the self-training
process, combined with the fusion of base samples in the memory bank. Extensive
experiments demonstrate a 53% improvement in novel recall across diverse OV
settings, VLMs, and 3D detectors. Notably, we achieve up to a 3.97-fold
increase in Average Precision (AP) for novel object classes. The source code is
made available in the supplementary material.

通过开放式词汇学习、视觉语言模型和远程模拟器，我们解决了当前基于激光雷达的 3D 物体检测系统的局限性，实现了对新颖对象的高效检测和识别。

在城市环境中的开放词汇三维物体检测

Find n' Propagate: Open-Vocabulary 3D Object Detection in Urban  Environments

Open-vocabulary learning has emerged as a cutting-edge research area,
particularly in light of the widespread adoption of vision-based foundational
models. Its primary objective is to comprehend novel concepts that are not
encompassed within a predefined vocabulary. One key facet of this endeavor is
Visual Grounding, which entails locating a specific region within an image
based on a corresponding language description. While current foundational
models excel at various visual language tasks, there's a noticeable absence of
models specifically tailored for open-vocabulary visual grounding. This
research endeavor introduces novel and challenging OV tasks, namely
Open-Vocabulary Visual Grounding and Open-Vocabulary Phrase Localization. The
overarching aim is to establish connections between language descriptions and
the localization of novel objects. To facilitate this, we have curated a
comprehensive annotated benchmark, encompassing 7,272 OV-VG images and 1,000
OV-PL images. In our pursuit of addressing these challenges, we delved into
various baseline methodologies rooted in existing open-vocabulary object
detection, VG, and phrase localization frameworks. Surprisingly, we discovered
that state-of-the-art methods often falter in diverse scenarios. Consequently,
we developed a novel framework that integrates two critical components:
Text-Image Query Selection and Language-Guided Feature Attention. These modules
are designed to bolster the recognition of novel categories and enhance the
alignment between visual and linguistic information. Extensive experiments
demonstrate the efficacy of our proposed framework, which consistently attains
SOTA performance across the OV-VG task. Additionally, ablation studies provide
further evidence of the effectiveness of our innovative models. Codes and
datasets will be made publicly available at this https URL

本研究的关键词是开放式词汇学习、视觉定位、新概念、语言描述和对象检测。通过引入新颖和具有挑战性的任务，即开放式视觉定位与开放式短语定位，这项研究的目标是在语言描述和新对象的定位之间建立联系。我们提出了一种新的框架来加强对新类别的识别，并增强视觉和语言信息之间的对齐。经过广泛的实验验证，我们的提议框架在开放式视觉定位任务上始终达到了最先进的性能。

OV-VG：开放词汇视觉定位基准

OV-VG: A Benchmark for Open-Vocabulary Visual Grounding

Recently, open-vocabulary learning has emerged to accomplish segmentation for
arbitrary categories of text-based descriptions, which popularizes the
segmentation system to more general-purpose application scenarios. However,
existing methods devote to designing specialized architectures or parameters
for specific segmentation tasks. These customized design paradigms lead to
fragmentation between various segmentation tasks, thus hindering the uniformity
of segmentation models. Hence in this paper, we propose FreeSeg, a generic
framework to accomplish Unified, Universal and Open-Vocabulary Image
Segmentation. FreeSeg optimizes an all-in-one network via one-shot training and
employs the same architecture and parameters to handle diverse segmentation
tasks seamlessly in the inference procedure. Additionally, adaptive prompt
learning facilitates the unified model to capture task-aware and
category-sensitive concepts, improving model robustness in multi-task and
varied scenarios. Extensive experimental results demonstrate that FreeSeg
establishes new state-of-the-art results in performance and generalization on
three segmentation tasks, which outperforms the best task-specific
architectures by a large margin: 5.5% mIoU on semantic segmentation, 17.6% mAP
on instance segmentation, 20.1% PQ on panoptic segmentation for the unseen
class on COCO.

本文提出了一种通用框架 FreeSeg，通过一次训练优化全一体网络，并采用相同的架构和参数在推理过程中无缝处理各种分割任务。此外，自适应提示学习有助于统一的模型捕捉任务感知和类别敏感概念，在多任务和各种场景中提高模型鲁棒性。大量实验结果表明，在三个分割任务上，FreeSeg 在性能和泛化方面建立了新的最先进结果，比最好的任务特定架构高出很大的幅度：在语义分割、实例分割和 COCO 上的全景分割中，未见类别的 PQ 分别达到 5.5％，17.6％和 20.1％。