As large language models (LLMs) evolve, their integration with 3D spatial
data (3D-LLMs) has seen rapid progress, offering unprecedented capabilities for
understanding and interacting with physical spaces. This survey provides a
comprehensive overview of the methodologies enabling LLMs to process,
understand, and generate 3D data. Highlighting the unique advantages of LLMs,
such as in-context learning, step-by-step reasoning, open-vocabulary
capabilities, and extensive world knowledge, we underscore their potential to
significantly advance spatial comprehension and interaction within embodied
Artificial Intelligence (AI) systems. Our investigation spans various 3D data
representations, from point clouds to Neural Radiance Fields (NeRFs). It
examines their integration with LLMs for tasks such as 3D scene understanding,
captioning, question-answering, and dialogue, as well as LLM-based agents for
spatial reasoning, planning, and navigation. The paper also includes a brief
review of other methods that integrate 3D and language. The meta-analysis
presented in this paper reveals significant progress yet underscores the
necessity for novel approaches to harness the full potential of 3D-LLMs. Hence,
with this paper, we aim to chart a course for future research that explores and
expands the capabilities of 3D-LLMs in understanding and interacting with the
complex 3D world. To support this survey, we have established a project page
where papers related to our topic are organized and listed:
this https URL

大型语言模型（LLMs）与三维空间数据（3D-LLMs）的整合不断进步，为理解和与物理空间互动提供了前所未有的能力。本文综述了使 LLMs 能够处理、理解和生成 3D 数据的方法，强调了其在上下文学习、逐步推理、开放词汇能力和广泛的世界知识等独特优势，并强调了它们在体现人工智能系统中显著促进空间理解和互动方面的潜力。我们的研究涵盖了从点云到神经辐射场（NeRFs）等各种 3D 数据表示，分析了它们与 LLMs 的整合在 3D 场景理解、字幕生成、问答和对话，以及基于 LLMs 的空间推理、规划和导航等任务中的应用，同时还对整合 3D 和语言的其他方法进行了简要回顾。本文的元分析显示取得了显著进展，但也强调了利用 3D-LLMs 的全部潜力需要创新方法的必要性。因此，本文旨在为未来的研究规划一个探索和扩展 3D-LLMs 在理解和与复杂 3D 世界互动方面能力的道路。为了支持这项综述，我们建立了一个项目页面，其中整理和列出了与我们的主题相关的论文。