The rising importance of 3D representation learning, pivotal in computer
vision, autonomous driving, and robotics, is evident. However, a prevailing
trend, which straightforwardly resorted to transferring 2D alignment strategies
to the 3D domain, encounters three distinct challenges: (1) Information
Degradation: This arises from the alignment of 3D data with mere single-view 2D
images and generic texts, neglecting the need for multi-view images and
detailed subcategory texts. (2) Insufficient Synergy: These strategies align 3D
representations to image and text features individually, hampering the overall
optimization for 3D models. (3) Underutilization: The fine-grained information
inherent in the learned representations is often not fully exploited,
indicating a potential loss in detail. To address these issues, we introduce
JM3D, a comprehensive approach integrating point cloud, text, and image. Key
contributions include the Structured Multimodal Organizer (SMO), enriching
vision-language representation with multiple views and hierarchical text, and
the Joint Multi-modal Alignment (JMA), combining language understanding with
visual representation. Our advanced model, JM3D-LLM, marries 3D representation
with large language models via efficient fine-tuning. Evaluations on ModelNet40
and ScanObjectNN establish JM3D's superiority. The superior performance of
JM3D-LLM further underscores the effectiveness of our representation transfer
approach. Our code and models are available at this https URL

为了解决 3D 数据与 2D 图像和文本的对齐问题带来的信息退化、协同不足和细节利用不充分的挑战，本文介绍了一种综合点云、文本和图像的综合方法 JM3D，其关键贡献是结构化多模态组织器和联合多模态对齐，通过高效的微调将 3D 表示与大型语言模型相结合，JM3D-LLM 在 ModelNet40 和 ScanObjectNN 的评估中表现出优势，进一步证明了我们的表示转换方法的有效性。