Large language models have become a potential pathway toward achieving
artificial general intelligence. Recent works on multi-modal large language
models have demonstrated their effectiveness in handling visual modalities. In
this work, we extend the research of MLLMs to point clouds and present the
LAMM-Dataset and LAMM-Benchmark for 2D image and 3D point cloud understanding.
We also establish an extensible framework to facilitate the extension of MLLMs
to additional modalities. Our main contribution is three-fold: 1) We present
the LAMM-Dataset and LAMM-Benchmark, which cover almost all high-level vision
tasks for 2D and 3D vision. Extensive experiments validate the effectiveness of
our dataset and benchmark. 2) We demonstrate the detailed methods of
constructing instruction-tuning datasets and benchmarks for MLLMs, which will
enable future research on MLLMs to scale up and extend to other domains, tasks,
and modalities faster. 3) We provide a primary but potential MLLM training
framework optimized for modalities' extension. We also provide baseline models,
comprehensive experimental observations, and analysis to accelerate future
research.

这篇研究将大型语言模型（MLLM）扩展到处理点云，建立了 LAMM 数据集和基准，为 MLLM 建立了可扩展的框架以适用于额外的模态，任务和领域。