As large language models (LLMs) evolve, their integration with 3D spatial
data (3D-LLMs) has seen rapid progress, offering unprecedented capabilities for
understanding and interacting with physical spaces. This survey provides a
comprehensive overview of the methodologies enabling LLMs to process,
understand, and generate 3D data. Highlighting the unique advantages of LLMs,
such as in-context learning, step-by-step reasoning, open-vocabulary
capabilities, and extensive world knowledge, we underscore their potential to
significantly advance spatial comprehension and interaction within embodied
Artificial Intelligence (AI) systems. Our investigation spans various 3D data
representations, from point clouds to Neural Radiance Fields (NeRFs). It
examines their integration with LLMs for tasks such as 3D scene understanding,
captioning, question-answering, and dialogue, as well as LLM-based agents for
spatial reasoning, planning, and navigation. The paper also includes a brief
review of other methods that integrate 3D and language. The meta-analysis
presented in this paper reveals significant progress yet underscores the
necessity for novel approaches to harness the full potential of 3D-LLMs. Hence,
with this paper, we aim to chart a course for future research that explores and
expands the capabilities of 3D-LLMs in understanding and interacting with the
complex 3D world. To support this survey, we have established a project page
where papers related to our topic are organized and listed:
this https URL

大型语言模型（LLMs）与三维空间数据（3D-LLMs）的整合不断进步，为理解和与物理空间互动提供了前所未有的能力。本文综述了使 LLMs 能够处理、理解和生成 3D 数据的方法，强调了其在上下文学习、逐步推理、开放词汇能力和广泛的世界知识等独特优势，并强调了它们在体现人工智能系统中显著促进空间理解和互动方面的潜力。我们的研究涵盖了从点云到神经辐射场（NeRFs）等各种 3D 数据表示，分析了它们与 LLMs 的整合在 3D 场景理解、字幕生成、问答和对话，以及基于 LLMs 的空间推理、规划和导航等任务中的应用，同时还对整合 3D 和语言的其他方法进行了简要回顾。本文的元分析显示取得了显著进展，但也强调了利用 3D-LLMs 的全部潜力需要创新方法的必要性。因此，本文旨在为未来的研究规划一个探索和扩展 3D-LLMs 在理解和与复杂 3D 世界互动方面能力的道路。为了支持这项综述，我们建立了一个项目页面，其中整理和列出了与我们的主题相关的论文。

LLMs 跨入 3D 世界：多模态大语言模型对 3D 任务的调查与元分析

When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks  via Multi-modal Large Language Models

We propose CompFuser, an image generation pipeline that enhances spatial
comprehension and attribute assignment in text-to-image generative models. Our
pipeline enables the interpretation of instructions defining spatial
relationships between objects in a scene, such as `An image of a gray cat on
the left of an orange dog', and generate corresponding images. This is
especially important in order to provide more control to the user. CompFuser
overcomes the limitation of existing text-to-image diffusion models by decoding
the generation of multiple objects into iterative steps: first generating a
single object and then editing the image by placing additional objects in their
designated positions. To create training data for spatial comprehension and
attribute assignment we introduce a synthetic data generation process, that
leverages a frozen large language model and a frozen layout-based diffusion
model for object placement. We compare our approach to strong baselines and
show that our model outperforms state-of-the-art image generation models in
spatial comprehension and attribute assignment, despite being 3x to 5x smaller
in parameters.

我们提出了 CompFuser，这是一个图像生成管道，可以增强文本到图像生成模型中的空间理解和属性分配。该管道能够解释定义场景中物体之间空间关系的指令，并生成相应的图像，为用户提供更多的控制。通过在生成多个对象时将其解码成迭代步骤：首先生成单个对象，然后通过将其他对象放置在指定位置来编辑图像，CompFuser 克服了现有文本到图像扩散模型的局限性。为了创建空间理解和属性分配的训练数据，我们引入了一种合成数据生成过程，利用冻结的大型语言模型和基于布局的扩散模型来进行对象放置。我们将我们的方法与强基准进行比较，并展示了我们的模型在空间理解和属性分配方面优于最先进的图像生成模型，尽管其参数只有现有模型的 3 倍到 5 倍小。