Evaluating the performance of Multi-modal Large Language Models (MLLMs),
integrating both point cloud and language, presents significant challenges. The
lack of a comprehensive assessment hampers determining whether these models
truly represent advancements, thereby impeding further progress in the field.
Current evaluations heavily rely on classification and caption tasks, falling
short in providing a thorough assessment of MLLMs. A pressing need exists for a
more sophisticated evaluation method capable of thoroughly analyzing the
spatial understanding and expressive capabilities of these models. To address
these issues, we introduce a scalable 3D benchmark, accompanied by a
large-scale instruction-tuning dataset known as 3DBench, providing an
extensible platform for a comprehensive evaluation of MLLMs. Specifically, we
establish the benchmark that spans a wide range of spatial and semantic scales,
from object-level to scene-level, addressing both perception and planning
tasks. Furthermore, we present a rigorous pipeline for automatically
constructing scalable 3D instruction-tuning datasets, covering 10 diverse
multi-modal tasks with more than 0.23 million QA pairs generated in total.
Thorough experiments evaluating trending MLLMs, comparisons against existing
datasets, and variations of training protocols demonstrate the superiority of
3DBench, offering valuable insights into current limitations and potential
research directions.

评估多模态大型语言模型（MLLMs）的性能，集成点云和语言，面临重大挑战。缺乏全面评估阻碍确定这些模型是否真正代表进步，从而妨碍该领域的进一步发展。我们引入了可扩展的 3D 基准测试和大规模指令调整数据集 3DBench，提供了一个全面评估 MLLMs 的可扩展平台，以解决这些问题。该基准测试覆盖了从物体级到场景级的广泛空间和语义范围，涉及感知和规划任务，并且我们提出了一个严格的流程来自动生成可扩展的 3D 指令调整数据集，总共涵盖了 10 个多模态任务，共生成了超过 23 万个问答对。通过对热门 MLLMs 的彻底实验评估、与现有数据集的比较以及训练方案的变异，我们展示了 3DBench 的优越性，为当前的局限性和潜在研究方向提供了有价值的见解。