Rapid advancements in continual segmentation have yet to bridge the gap of
scaling to large continually expanding vocabularies under compute-constrained
scenarios. We discover that traditional continual training leads to
catastrophic forgetting under compute constraints, unable to outperform
zero-shot segmentation methods. We introduce a novel strategy for semantic and
panoptic segmentation with zero forgetting, capable of adapting to continually
growing vocabularies without the need for retraining or large memory costs. Our
training-free approach, kNN-CLIP, leverages a database of instance embeddings
to enable open-vocabulary segmentation approaches to continually expand their
vocabulary on any given domain with a single-pass through data, while only
storing embeddings minimizing both compute and memory costs. This method
achieves state-of-the-art mIoU performance across large-vocabulary semantic and
panoptic segmentation datasets. We hope kNN-CLIP represents a step forward in
enabling more efficient and adaptable continual segmentation, paving the way
for advances in real-world large-vocabulary continual segmentation methods.

我们引入了一种新颖的连续语义和全景分割策略，名为 kNN-CLIP，它能够在计算和内存成本最小化的情况下，实现对不断扩大的词汇的适应，并在大词汇语义和全景分割数据集上取得了最先进的性能表现，为实现更高效、可适应性强的连续分割方法迈出了一步。

kNN-CLIP: 基于检索的训练免费的连续扩展大词汇的分割

kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually  Expanding Large Vocabularies

Most continual segmentation methods tackle the problem as a per-pixel
classification task. However, such a paradigm is very challenging, and we find
query-based segmenters with built-in objectness have inherent advantages
compared with per-pixel ones, as objectness has strong transfer ability and
forgetting resistance. Based on these findings, we propose CoMasTRe by
disentangling continual segmentation into two stages: forgetting-resistant
continual objectness learning and well-researched continual classification.
CoMasTRe uses a two-stage segmenter learning class-agnostic mask proposals at
the first stage and leaving recognition to the second stage. During continual
learning, a simple but effective distillation is adopted to strengthen
objectness. To further mitigate the forgetting of old classes, we design a
multi-label class distillation strategy suited for segmentation. We assess the
effectiveness of CoMasTRe on PASCAL VOC and ADE20K. Extensive experiments show
that our method outperforms per-pixel and query-based methods on both datasets.
Code will be available at this https URL

基于强大的目标性和防遗忘特性，我们提出了 CoMasTRe 方法，它将持续分割任务分为两个阶段，通过学习抗遗忘的目标性特征和经过良好研究的分类特征，采用简单但有效的蒸馏策略来强化目标性，并通过多标签类蒸馏策略减轻旧类别的遗忘，实验证明该方法在 PASCAL VOC 数据集上优于像素级和基于查询的方法。