Recent research has demonstrated that Large Language Models (LLMs) are not limited to text-only tasks but can also function as multimodal models across various modalities, including audio, images, and videos. In particular, research on 3D Large Multimodal Models (3D LMMs) is making notable strides, driven by the potential of processing higher-dimensional data like point clouds. However, upon closer examination, we find that the visual and textual content within each sample of existing training datasets lacks both high informational granularity and clarity, which serve as a bottleneck for precise cross-modal understanding. To address these issues, we propose CL3DOR, Contrastive Learning for 3D large multimodal models via Odds ratio on high-Resolution point clouds, designed to ensure greater specificity and clarity in both visual and textual content. Specifically, we increase the density of point clouds per object and construct informative hard negative responses in the training dataset to penalize unwanted responses. To leverage hard negative responses, we incorporate the odds ratio as an auxiliary term for contrastive learning into the conventional language modeling loss. CL3DOR achieves state-of-the-art performance in 3D scene understanding and reasoning benchmarks. Additionally, we demonstrate the effectiveness of CL3DOR's key components through extensive experiments.

本研究针对现有训练数据集中视觉和文本内容信息颗粒度不足和清晰度不高的问题，提出了CL3DOR模型，通过增加点云密度和构建信息丰富的困难负样本来提升多模态理解的精准性与清晰度。实验结果表明，CL3DOR在3D场景理解和推理基准测试中达到了最先进的表现，展示了其关键组件的有效性。

CL3DOR：通过高分辨率点云的赔率比进行3D大规模多模态模型的对比学习