Grounding natural language to the physical world is a ubiquitous topic with a
wide range of applications in computer vision and robotics. Recently, 2D
vision-language models such as CLIP have been widely popularized, due to their
impressive capabilities for open-vocabulary grounding in 2D images. Recent
works aim to elevate 2D CLIP features to 3D via feature distillation, but
either learn neural fields that are scene-specific and hence lack
generalization, or focus on indoor room scan data that require access to
multiple camera views, which is not practical in robot manipulation scenarios.
Additionally, related methods typically fuse features at pixel-level and assume
that all camera views are equally informative. In this work, we show that this
approach leads to sub-optimal 3D features, both in terms of grounding accuracy,
as well as segmentation crispness. To alleviate this, we propose a multi-view
feature fusion strategy that employs object-centric priors to eliminate
uninformative views based on semantic information, and fuse features at
object-level via instance segmentation masks. To distill our object-centric 3D
features, we generate a large-scale synthetic multi-view dataset of cluttered
tabletop scenes, spawning 15k scenes from over 3300 unique object instances,
which we make publicly available. We show that our method reconstructs 3D CLIP
features with improved grounding capacity and spatial consistency, while doing
so from single-view RGB-D, thus departing from the assumption of multiple
camera views at test time. Finally, we show that our approach can generalize to
novel tabletop domains and be re-purposed for 3D instance segmentation without
fine-tuning, and demonstrate its utility for language-guided robotic grasping
in clutter

将自然语言与物理世界联系起来是一个应用广泛的话题，该论文提出了一种基于对象中心先验知识的多视角特征融合策略，用于改善基于 2D 和 3D 图像的自然语言 grounding 和语言引导机器人抓取任务。

3D 特征蒸馏与对象为中心先验

3D Feature Distillation with Object-Centric Priors

Zero-shot 3D point cloud understanding can be achieved via 2D Vision-Language
Models (VLMs). Existing strategies directly map Vision-Language Models from 2D
pixels of rendered or captured views to 3D points, overlooking the inherent and
expressible point cloud geometric structure. Geometrically similar or close
regions can be exploited for bolstering point cloud understanding as they are
likely to share semantic information. To this end, we introduce the first
training-free aggregation technique that leverages the point cloud's 3D
geometric structure to improve the quality of the transferred Vision-Language
Models. Our approach operates iteratively, performing local-to-global
aggregation based on geometric and semantic point-level reasoning. We benchmark
our approach on three downstream tasks, including classification, part
segmentation, and semantic segmentation, with a variety of datasets
representing both synthetic/real-world, and indoor/outdoor scenarios. Our
approach achieves new state-of-the-art results in all benchmarks. We will
release the source code publicly.

通过利用点云的几何结构改进传输的视觉 - 语言模型，我们提出了首个无需训练的聚合技术，通过几何和语义的点级推理，在分类、部件分割和语义分割等三个任务中取得了新的技术优势。