Open-vocabulary semantic segmentation aims at segmenting arbitrary categories
expressed in textual form. Previous works have trained over large amounts of
image-caption pairs to enforce pixel-level multimodal alignments. However,
captions provide global information about the semantics of a given image but
lack direct localization of individual concepts. Further, training on
large-scale datasets inevitably brings significant computational costs. In this
paper, we propose FreeDA, a training-free diffusion-augmented method for
open-vocabulary semantic segmentation, which leverages the ability of diffusion
models to visually localize generated concepts and local-global similarities to
match class-agnostic regions with semantic classes. Our approach involves an
offline stage in which textual-visual reference embeddings are collected,
starting from a large set of captions and leveraging visual and semantic
contexts. At test time, these are queried to support the visual matching
process, which is carried out by jointly considering class-agnostic regions and
global semantic similarities. Extensive analyses demonstrate that FreeDA
achieves state-of-the-art performance on five datasets, surpassing previous
methods by more than 7.0 average points in terms of mIoU and without requiring
any training.

该论文介绍了一种用于开放词汇语义分割的无需训练的扩散增强方法（FreeDA），通过扩散模型的能力来可视化生成概念并通过局部 - 全局相似性将无类别区域与语义类别匹配，从而在五个数据集上实现了超过 7.0 平均 IoU 点的最先进性能，而无需任何训练。