3D Shape represented as point cloud has achieve advancements in multimodal pre-training to align image and language descriptions, which is curial to object identification, classification, and retrieval. However, the discrete representations of point cloud lost the object's surface shape information and creates a gap between rendering results and 2D correspondences. To address this problem, we propose GS-CLIP for the first attempt to introduce 3DGS (3D Gaussian Splatting) into multimodal pre-training to enhance 3D representation. GS-CLIP leverages a pre-trained vision-language model for a learned common visual and textual space on massive real world image-text pairs and then learns a 3D Encoder for aligning 3DGS optimized per object. Additionally, a novel Gaussian-Aware Fusion is proposed to extract and fuse global explicit feature. As a general framework for language-image-3D pre-training, GS-CLIP is agnostic to 3D backbone networks. Experiments on challenging shows that GS-CLIP significantly improves the state-of-the-art, outperforming the previously best results.

提出GS-CLIP方法将3D高斯粒化引入多模态预训练，以增强3D表示，并通过预训练的视觉-语言模型和3D编码器对3D高斯粒化进行优化，实现全局显式特征的提取和融合。实验证明，GS-CLIP显著提高了最新研究成果，超越了先前最好的结果。

GS-CLIP: 从真实世界数据进行对比度语言-图像-3D预训练的高斯飞溅