The superior performances of pre-trained foundation models in various visual
tasks underscore their potential to enhance the 2D models' open-vocabulary
ability. Existing methods explore analogous applications in the 3D space.
However, most of them only center around knowledge extraction from singular
foundation models, which limits the open-vocabulary ability of 3D models. We
hypothesize that leveraging complementary pre-trained knowledge from various
foundation models can improve knowledge transfer from 2D pre-trained visual
language models to the 3D space. In this work, we propose FM-OV3D, a method of
Foundation Model-based Cross-modal Knowledge Blending for Open-Vocabulary 3D
Detection, which improves the open-vocabulary localization and recognition
abilities of 3D model by blending knowledge from multiple pre-trained
foundation models, achieving true open-vocabulary without facing constraints
from original 3D datasets. Specifically, to learn the open-vocabulary 3D
localization ability, we adopt the open-vocabulary localization knowledge of
the Grounded-Segment-Anything model. For open-vocabulary 3D recognition
ability, We leverage the knowledge of generative foundation models, including
GPT-3 and Stable Diffusion models, and cross-modal discriminative models like
CLIP. The experimental results on two popular benchmarks for open-vocabulary 3D
object detection show that our model efficiently learns knowledge from multiple
foundation models to enhance the open-vocabulary ability of the 3D model and
successfully achieves state-of-the-art performance in open-vocabulary 3D object
detection tasks. Code is released at
this https URL

基于各种基础模型的跨模态知识融合方法可以提高 3D 模型的开放性词汇能力，并成功实现了在开放性 3D 目标检测任务中领先的性能。

FM-OV3D：面向开放词汇的基于基础模型的跨模态知识融合的 3D 目标检测

FM-OV3D: Foundation Model-based Cross-modal Knowledge Blending for  Open-Vocabulary 3D Detection

Traditional LiDAR-based object detection research primarily focuses on
closed-set scenarios, which falls short in complex real-world applications.
Directly transferring existing 2D open-vocabulary models with some known LiDAR
classes for open-vocabulary ability, however, tends to suffer from over-fitting
problems: The obtained model will detect the known objects, even presented with
a novel category. In this paper, we propose OpenSight, a more advanced 2D-3D
modeling framework for LiDAR-based open-vocabulary detection. OpenSight
utilizes 2D-3D geometric priors for the initial discernment and localization of
generic objects, followed by a more specific semantic interpretation of the
detected objects. The process begins by generating 2D boxes for generic objects
from the accompanying camera images of LiDAR. These 2D boxes, together with
LiDAR points, are then lifted back into the LiDAR space to estimate
corresponding 3D boxes. For better generic object perception, our framework
integrates both temporal and spatial-aware constraints. Temporal awareness
correlates the predicted 3D boxes across consecutive timestamps, recalibrating
the missed or inaccurate boxes. The spatial awareness randomly places some
``precisely'' estimated 3D boxes at varying distances, increasing the
visibility of generic objects. To interpret the specific semantics of detected
objects, we develop a cross-modal alignment and fusion module to first align 3D
features with 2D image embeddings and then fuse the aligned 3D-2D features for
semantic decoding. Our experiments indicate that our method establishes
state-of-the-art open-vocabulary performance on widely used 3D detection
benchmarks and effectively identifies objects for new categories of interest.

提出了基于 LiDAR 的开放词汇检测的先进 2D-3D 建模框架 OpenSight，通过生成 2D 边框并将其与 LiDAR 点云转换为对应的 3D 边框，增强了通用对象感知能力，并通过交叉模态对齐和融合进行特定语义解码，从而在广泛应用的 3D 检测基准上建立了最先进的开放词汇性能，并有效识别感兴趣的新类别对象。