Gaussian Splatting has revolutionized the world of novel view synthesis by
achieving high rendering performance in real-time. Recently, studies have
focused on enriching these 3D representations with semantic information for
downstream tasks. In this paper, we introduce RT-GS2, the first generalizable
semantic segmentation method employing Gaussian Splatting. While existing
Gaussian Splatting-based approaches rely on scene-specific training, RT-GS2
demonstrates the ability to generalize to unseen scenes. Our method adopts a
new approach by first extracting view-independent 3D Gaussian features in a
self-supervised manner, followed by a novel View-Dependent / View-Independent
(VDVI) feature fusion to enhance semantic consistency over different views.
Extensive experimentation on three different datasets showcases RT-GS2's
superiority over the state-of-the-art methods in semantic segmentation quality,
exemplified by a 8.01% increase in mIoU on the Replica dataset. Moreover, our
method achieves real-time performance of 27.03 FPS, marking an astonishing 901
times speedup compared to existing approaches. This work represents a
significant advancement in the field by introducing, to the best of our
knowledge, the first real-time generalizable semantic segmentation method for
3D Gaussian representations of radiance fields.

RT-GS2 是第一个能够实现实时泛化的 3D 高斯渲染中语义分割方法，通过提取独立于视角的 3D 高斯特征并采用新颖的视角相关 / 独立特征融合，能够在不同视角下提升语义一致性，超过同领域最先进方法的语义分割质量达到 8.01％的提高，并以每秒 27.03 帧的实时性能，相比现有方法提速达 901 倍，这一工作在引入光辐射场 3D 高斯表达领域的语义分割上代表了重大进展。