While 3D MLLMs have achieved significant progress, they are restricted to
object and scene understanding and struggle to understand 3D spatial structures
at the part level. In this paper, we introduce Kestrel, representing a novel
approach that empowers 3D MLLMs with part-aware understanding, enabling better
interpretation and segmentation grounding of 3D objects at the part level.
Despite its significance, the current landscape lacks tasks and datasets that
endow and assess this capability. Therefore, we propose two novel tasks: (1)
Part-Aware Point Grounding, the model is tasked with directly predicting a
part-level segmentation mask based on user instructions, and (2) Part-Aware
Point Grounded Captioning, the model provides a detailed caption that includes
part-level descriptions and their corresponding masks. To support learning and
evaluating for these tasks, we introduce 3DCoMPaT Grounded Instructions Dataset
(3DCoMPaT-GRIN). 3DCoMPaT-GRIN Vanilla, comprising 789k part-aware point
cloud-instruction-segmentation mask triplets, is used to evaluate MLLMs'
ability of part-aware segmentation grounding. 3DCoMPaT-GRIN Grounded Caption,
containing 107k part-aware point cloud-instruction-grounded caption triplets,
assesses both MLLMs' part-aware language comprehension and segmentation
grounding capabilities. Our introduced tasks, dataset, and Kestrel represent a
preliminary effort to bridge the gap between human cognition and 3D MLLMs,
i.e., the ability to perceive and engage with the environment at both global
and part levels. Extensive experiments on the 3DCoMPaT-GRIN show that Kestrel
can generate user-specified segmentation masks, a capability not present in any
existing 3D MLLM. Kestrel thus established a benchmark for evaluating the
part-aware language comprehension and segmentation grounding of 3D objects.
Project page at this https URL

Kestrel 是一种新颖的方法，为 3D MLLMs 赋予了部分感知的理解能力，并能够更好地解释和分割 3D 对象的各个部分。研究提出了两项新任务：局部感知点定位和局部感知点定位的字幕生成，并引入了用于学习和评估这些任务的数据集 3DCoMPaT-GRIN。实验证明 Kestrel 能够生成用户指定的分割掩码，这在现有的 3D MLLM 中尚不具备。因此，Kestrel 为评估 3D 对象的部分感知语言理解和分割定位能力建立了基准。