We introduce Point-Bind, a 3D multi-modality model aligning point clouds with
2D image, language, audio, and video. Guided by ImageBind, we construct a joint
embedding space between 3D and multi-modalities, enabling many promising
applications, e.g., any-to-3D generation, 3D embedding arithmetic, and 3D
open-world understanding. On top of this, we further present Point-LLM, the
first 3D large language model (LLM) following 3D multi-modal instructions. By
parameter-efficient fine-tuning techniques, Point-LLM injects the semantics of
Point-Bind into pre-trained LLMs, e.g., LLaMA, which requires no 3D instruction
data, but exhibits superior 3D and multi-modal question-answering capacity. We
hope our work may cast a light on the community for extending 3D point clouds
to multi-modality applications. Code is available at
this https URL

我们引入了 Point-Bind，这是一个将三维点云与二维图像、语言、音频和视频对齐的多模态模型。通过 ImageBind 的引导，我们构建了三维和多模态之间的联合嵌入空间，实现了许多有前景的应用，例如任意到三维生成、三维嵌入算术和三维开放世界理解。此外，我们还提出了 Point-LLM，这是第一个遵循三维多模态指令的三维大型语言模型（LLM）。通过参数高效的微调技术，Point-LLM 将 Point-Bind 的语义注入到预训练的 LLM 中，例如 LLaMA，它不需要三维指令数据，但具有出色的三维和多模态问答能力。我们希望我们的工作可以为扩展三维点云到多模态应用的社区照亮一线。