Thanks to the impressive progress of large-scale vision-language pretraining,
recent recognition models can classify arbitrary objects in a zero-shot and
open-set manner, with a surprisingly high accuracy. However, translating this
success to semantic segmentation is not trivial, because this dense prediction
task requires not only accurate semantic understanding but also fine shape
delineation and existing vision-language models are trained with image-level
language descriptions. To bridge this gap, we pursue \textbf{shape-aware}
zero-shot semantic segmentation in this study. Inspired by classical spectral
methods in the image segmentation literature, we propose to leverage the eigen
vectors of Laplacian matrices constructed with self-supervised pixel-wise
features to promote shape-awareness. Despite that this simple and effective
technique does not make use of the masks of seen classes at all, we demonstrate
that it out-performs a state-of-the-art shape-aware formulation that aligns
ground truth and predicted edges during training. We also delve into the
performance gains achieved on different datasets using different backbones and
draw several interesting and conclusive observations: the benefits of promoting
shape-awareness highly relates to mask compactness and language embedding
locality. Finally, our method sets new state-of-the-art performance for
zero-shot semantic segmentation on both Pascal and COCO, with significant
margins. Code and models will be accessed at this https URL

本研究通过使用基于自监督像素级特征构建的拉普拉斯矩阵的特征向量提高了形状感知的零样本语义分割准确性，实现了比现有成果更好的零样本语义分割表现。