As large language models (LLMs) evolve, their integration with 3D spatial
data (3D-LLMs) has seen rapid progress, offering unprecedented capabilities for
understanding and interacting with physical spaces. This survey provides a
comprehensive overview of the methodologies enabling LLMs to process,
understand, and generate 3D data. Highlighting the unique advantages of LLMs,
such as in-context learning, step-by-step reasoning, open-vocabulary
capabilities, and extensive world knowledge, we underscore their potential to
significantly advance spatial comprehension and interaction within embodied
Artificial Intelligence (AI) systems. Our investigation spans various 3D data
representations, from point clouds to Neural Radiance Fields (NeRFs). It
examines their integration with LLMs for tasks such as 3D scene understanding,
captioning, question-answering, and dialogue, as well as LLM-based agents for
spatial reasoning, planning, and navigation. The paper also includes a brief
review of other methods that integrate 3D and language. The meta-analysis
presented in this paper reveals significant progress yet underscores the
necessity for novel approaches to harness the full potential of 3D-LLMs. Hence,
with this paper, we aim to chart a course for future research that explores and
expands the capabilities of 3D-LLMs in understanding and interacting with the
complex 3D world. To support this survey, we have established a project page
where papers related to our topic are organized and listed:
this https URL

大型语言模型（LLMs）与三维空间数据（3D-LLMs）的整合不断进步，为理解和与物理空间互动提供了前所未有的能力。本文综述了使 LLMs 能够处理、理解和生成 3D 数据的方法，强调了其在上下文学习、逐步推理、开放词汇能力和广泛的世界知识等独特优势，并强调了它们在体现人工智能系统中显著促进空间理解和互动方面的潜力。我们的研究涵盖了从点云到神经辐射场（NeRFs）等各种 3D 数据表示，分析了它们与 LLMs 的整合在 3D 场景理解、字幕生成、问答和对话，以及基于 LLMs 的空间推理、规划和导航等任务中的应用，同时还对整合 3D 和语言的其他方法进行了简要回顾。本文的元分析显示取得了显著进展，但也强调了利用 3D-LLMs 的全部潜力需要创新方法的必要性。因此，本文旨在为未来的研究规划一个探索和扩展 3D-LLMs 在理解和与复杂 3D 世界互动方面能力的道路。为了支持这项综述，我们建立了一个项目页面，其中整理和列出了与我们的主题相关的论文。

LLMs 跨入 3D 世界：多模态大语言模型对 3D 任务的调查与元分析

When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks  via Multi-modal Large Language Models

Large 2D vision-language models (2D-LLMs) have gained significant attention
by bridging Large Language Models (LLMs) with images using a simple projector.
Inspired by their success, large 3D point cloud-language models (3D-LLMs) also
integrate point clouds into LLMs. However, directly aligning point clouds with
LLM requires expensive training costs, typically in hundreds of GPU-hours on
A100, which hinders the development of 3D-LLMs. In this paper, we introduce
MiniGPT-3D, an efficient and powerful 3D-LLM that achieves multiple SOTA
results while training for only 27 hours on one RTX 3090. Specifically, we
propose to align 3D point clouds with LLMs using 2D priors from 2D-LLMs, which
can leverage the similarity between 2D and 3D visual information. We introduce
a novel four-stage training strategy for modality alignment in a cascaded way,
and a mixture of query experts module to adaptively aggregate features with
high efficiency. Moreover, we utilize parameter-efficient fine-tuning methods
LoRA and Norm fine-tuning, resulting in only 47.8M learnable parameters, which
is up to 260x fewer than existing methods. Extensive experiments show that
MiniGPT-3D achieves SOTA on 3D object classification and captioning tasks, with
significantly cheaper training costs. Notably, MiniGPT-3D gains an 8.12
increase on GPT-4 evaluation score for the challenging object captioning task
compared to ShapeLLM-13B, while the latter costs 160 total GPU-hours on 8 A800.
We are the first to explore the efficient 3D-LLM, offering new insights to the
community. Code and weights are available at
this https URL

MiniGPT-3D 是一种高效而强大的 3D-LLM，仅经过 27 小时在一台 RTX 3090 上训练，在 2D-LLMs 的 2D 先验知识的帮助下，使用四阶段训练策略进行模态对齐，并采用高效聚合特征的混合查询专家模块，以及低参数的微调方法 LoRA 和 Norm 微调，在 3D 对象分类和字幕任务上实现了具有显著较低训练成本的 SOTA 结果。

MiniGPT-3D：利用 2D 先验信息高效对齐大型语言模型中的 3D 点云

MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language  Models using 2D Priors

Large language models (LLMs) and Vision-Language Models (VLMs) have been
proven to excel at multiple tasks, such as commonsense reasoning. Powerful as
these models can be, they are not grounded in the 3D physical world, which
involves richer concepts such as spatial relationships, affordances, physics,
layout, and so on. In this work, we propose to inject the 3D world into large
language models and introduce a whole new family of 3D-LLMs. Specifically,
3D-LLMs can take 3D point clouds and their features as input and perform a
diverse set of 3D-related tasks, including captioning, dense captioning, 3D
question answering, task decomposition, 3D grounding, 3D-assisted dialog,
navigation, and so on. Using three types of prompting mechanisms that we
design, we are able to collect over 300k 3D-language data covering these tasks.
To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that
obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as
our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism,
3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show
that our model outperforms state-of-the-art baselines by a large margin (e.g.,
the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore,
experiments on our held-in datasets for 3D captioning, task composition, and
3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative
examples also show that our model could perform more tasks beyond the scope of
existing LLMs and VLMs. Project Page: : this https URL

我们提出了一种新的 3D-LLMs，将 3D 世界引入大型语言模型，并利用 3D point clouds 进行各种 3D 相关任务，通过我们设计的提示机制收集了超过 300k 的 3D 语言数据，实验证明我们的模型在 ScanQA 上的表现超过了最先进的基准方法，并在 3D 字幕、任务组合和 3D 辅助对话方面优于 2D VLMs。