We introduce Gaga, a framework that reconstructs and segments open-world 3D
scenes by leveraging inconsistent 2D masks predicted by zero-shot segmentation
models. Contrasted to prior 3D scene segmentation approaches that heavily rely
on video object tracking, Gaga utilizes spatial information and effectively
associates object masks across diverse camera poses. By eliminating the
assumption of continuous view changes in training images, Gaga demonstrates
robustness to variations in camera poses, particularly beneficial for sparsely
sampled images, ensuring precise mask label consistency. Furthermore, Gaga
accommodates 2D segmentation masks from diverse sources and demonstrates robust
performance with different open-world zero-shot segmentation models, enhancing
its versatility. Extensive qualitative and quantitative evaluations demonstrate
that Gaga performs favorably against state-of-the-art methods, emphasizing its
potential for real-world applications such as scene understanding and
manipulation.

Gaga 是一个利用不一致的零样本分割模型预测的 2D 掩码来重建和分割开放环境下的 3D 场景的框架。通过利用空间信息，有效地关联不同摄像机姿势下的物体掩码，Gaga 消除了训练图像中关于视角连续变化的假设，展现出对摄像机姿势变化的鲁棒性，尤其在稀疏采样图像中，确保准确的掩码标签一致性。此外，Gaga 适应来自不同来源的 2D 分割掩码，并展现出与不同开放世界零样本分割模型的鲁棒性，提高了其通用性。广泛的定性和定量评估表明，Gaga 在与最先进的方法相比中表现出色，强调其在场景理解和操作等实际应用中的潜力。