Recent advancements in 3D perception systems have significantly improved
their ability to perform visual recognition tasks such as segmentation.
However, these systems still heavily rely on explicit human instruction to
identify target objects or categories, lacking the capability to actively
reason and comprehend implicit user intentions. We introduce a novel
segmentation task known as reasoning part segmentation for 3D objects, aiming
to output a segmentation mask based on complex and implicit textual queries
about specific parts of a 3D object. To facilitate evaluation and benchmarking,
we present a large 3D dataset comprising over 60k instructions paired with
corresponding ground-truth part segmentation annotations specifically curated
for reasoning-based 3D part segmentation. We propose a model that is capable of
segmenting parts of 3D objects based on implicit textual queries and generating
natural language explanations corresponding to 3D object segmentation requests.
Experiments show that our method achieves competitive performance to models
that use explicit queries, with the additional abilities to identify part
concepts, reason about them, and complement them with world knowledge. Our
source code, dataset, and trained models are available at
this https URL

鉴于目标物体识别需求，我们介绍了一种称为 3D 物体推理部分分割的新型细分任务，旨在根据复杂和隐含的文本查询输出一个基于 3D 物体特定部分的分割蒙版，并提出了一种能够分割 3D 物体部分的模型，并生成与 3D 物体分割请求相对应的自然语言解释。

基于推理的大型多模态模型的 3D 零件分割

PARIS3D: Reasoning-based 3D Part Segmentation Using Large Multimodal  Model

Low-cost, vision-centric 3D perception systems for autonomous driving have
made significant progress in recent years, narrowing the gap to expensive
LiDAR-based methods. The primary challenge in becoming a fully reliable
alternative lies in robust depth prediction capabilities, as camera-based
systems struggle with long detection ranges and adverse lighting and weather
conditions. In this work, we introduce HyDRa, a novel camera-radar fusion
architecture for diverse 3D perception tasks. Building upon the principles of
dense BEV (Bird's Eye View)-based architectures, HyDRa introduces a hybrid
fusion approach to combine the strengths of complementary camera and radar
features in two distinct representation spaces. Our Height Association
Transformer module leverages radar features already in the perspective view to
produce more robust and accurate depth predictions. In the BEV, we refine the
initial sparse representation by a Radar-weighted Depth Consistency. HyDRa
achieves a new state-of-the-art for camera-radar fusion of 64.2 NDS (+1.8) and
58.4 AMOTA (+1.5) on the public nuScenes dataset. Moreover, our new
semantically rich and spatially accurate BEV features can be directly converted
into a powerful occupancy representation, beating all previous camera-based
methods on the Occ3D benchmark by an impressive 3.7 mIoU.

通过融合摄像头和雷达特征的新型摄像头 - 雷达融合架构 HyDRa，在更广泛的 3D 感知任务中实现了新的最先进性能，并能将 BEV 特征直接转换为强大的占据表示，在 Occ3D 基准测试中超过以往基于摄像头的方法。