We propose SemGauss-SLAM, the first semantic SLAM system utilizing 3D
Gaussian representation, that enables accurate 3D semantic mapping, robust
camera tracking, and high-quality rendering in real-time. In this system, we
incorporate semantic feature embedding into 3D Gaussian representation, which
effectively encodes semantic information within the spatial layout of the
environment for precise semantic scene representation. Furthermore, we propose
feature-level loss for updating 3D Gaussian representation, enabling
higher-level guidance for 3D Gaussian optimization. In addition, to reduce
cumulative drift and improve reconstruction accuracy, we introduce
semantic-informed bundle adjustment leveraging semantic associations for joint
optimization of 3D Gaussian representation and camera poses, leading to more
robust tracking and consistent mapping. Our SemGauss-SLAM method demonstrates
superior performance over existing dense semantic SLAM methods in terms of
mapping and tracking accuracy on Replica and ScanNet datasets, while also
showing excellent capabilities in novel-view semantic synthesis and 3D semantic
mapping.

提出 SemGauss-SLAM，第一个利用 3D 高斯表示的语义 SLAM 系统，实现实时精确的 3D 语义地图构建、鲁棒的相机跟踪和高质量渲染。该系统将语义特征嵌入 3D 高斯表示，有效地在环境的空间布局中编码语义信息以获得精确的语义场景表示；同时，提出特征级损失来更新 3D 高斯表示，为 3D 高斯优化提供更高层次的指导；此外，通过引入语义感知束调整，利用语义关联进行 3D 高斯表示和相机位姿的联合优化，从而减小累积漂移并提高重建精度。在 Replica 和 ScanNet 数据集上，与现有的密集语义 SLAM 方法相比，我们的 SemGauss-SLAM 方法在地图构建和相机跟踪的准确性方面表现出更好的性能，并展示出新视角语义合成和 3D 语义映射的优秀能力。