Large-scale vision foundation models such as Segment Anything (SAM)
demonstrate impressive performance in zero-shot image segmentation at multiple
levels of granularity. However, these zero-shot predictions are rarely
3D-consistent. As the camera viewpoint changes in a scene, so do the
segmentation predictions, as well as the characterizations of ``coarse" or
``fine" granularity. In this work, we address the challenging task of lifting
multi-granular and view-inconsistent image segmentations into a hierarchical
and 3D-consistent representation. We learn a novel feature field within a
Neural Radiance Field (NeRF) representing a 3D scene, whose segmentation
structure can be revealed at different scales by simply using different
thresholds on feature distance. Our key idea is to learn an ultrametric feature
space, which unlike a Euclidean space, exhibits transitivity in distance-based
grouping, naturally leading to a hierarchical clustering. Put together, our
method takes view-inconsistent multi-granularity 2D segmentations as input and
produces a hierarchy of 3D-consistent segmentations as output. We evaluate our
method and several baselines on synthetic datasets with multi-view images and
multi-granular segmentation, showcasing improved accuracy and
viewpoint-consistency. We additionally provide qualitative examples of our
model's 3D hierarchical segmentations in real world scenes.\footnote{The code
and dataset are available at:

通过使用 Neural Radiance Field 中的新颖特征场，本研究提出了一种将多粒度、视角不一致的图像分割转化为层次化、3D 一致的表示的方法。通过在不同距离阈值下观察特征距离，可以揭示不同尺度下的分段结构。将视角不一致的多粒度 2D 分割作为输入，生成层次化的 3D 一致分割作为输出，展示了在合成数据集上准确性和视角一致性上的改进。并提供了模型在真实场景中生成的 3D 层次化分割的定性实例。