Precisely perceiving the geometric and semantic properties of real-world 3D
objects is crucial for the continued evolution of augmented reality and robotic
applications. To this end, we present \algfull{} (\algname{}), which
incorporates vision-language embeddings of foundation models into 3D Gaussian
Splatting (GS). The key contribution of this work is an efficient method to
reconstruct and represent 3D vision-language models. This is achieved by
distilling feature maps generated from image-based foundation models into those
rendered from our 3D model. To ensure high-quality rendering and fast training,
we introduce a novel scene representation by integrating strengths from both GS
and multi-resolution hash encodings (MHE). Our effective training procedure
also introduces a pixel alignment loss that makes the rendered feature distance
of same semantic entities close, following the pixel-level semantic boundaries.
Our results demonstrate remarkable multi-view semantic consistency,
facilitating diverse downstream tasks, beating state-of-the-art methods by
$\mathbf{10.2}$ percent on open-vocabulary language-based object detection,
despite that we are $\mathbf{851\times}$ faster for inference. This research
explores the intersection of vision, language, and 3D scene representation,
paving the way for enhanced scene understanding in uncontrolled real-world
environments. We plan to release the code upon paper acceptance.

通过将视觉语言嵌入模型与三维高斯喷洒算法相结合，本研究提出了一种高效的三维视觉语言模型重建方法，通过渲染从三维模型生成的特征映射，实现了高质量的渲染和快速训练，并引入了像素对齐损失来保证渲染的语义一致性，取得了显著的多视图语义一致性，为增强对无控制的现实环境中场景的理解铺平了道路。