Recent advances in Large Multimodal Models (LMM) have made it possible for
various applications in human-machine interactions. However, developing LMMs
that can comprehend, reason, and plan in complex and diverse 3D environments
remains a challenging topic, especially considering the demand for
understanding permutation-invariant point cloud 3D representations of the 3D
scene. Existing works seek help from multi-view images, and project 2D features
to 3D space as 3D scene representations. This, however, leads to huge
computational overhead and performance degradation. In this paper, we present
LL3DA, a Large Language 3D Assistant that takes point cloud as direct input and
respond to both textual-instructions and visual-prompts. This help LMMs better
comprehend human interactions and further help to remove the ambiguities in
cluttered 3D scenes. Experiments show that LL3DA achieves remarkable results,
and surpasses various 3D vision-language models on both 3D Dense Captioning and
3D Question Answering.

LL3DA 是一种大型语言 3D 助手，可以直接接收点云输入，并对文本指令和视觉提示进行回应，可以帮助大型多模态模型更好地理解人类互动，并在杂乱的 3D 场景中消除歧义。实验证明，LL3DA 在 3D 密集字幕和 3D 问答方面取得了显著成果，并超越了各种 3D 视觉语言模型。