Rapid advancements in continual segmentation have yet to bridge the gap of
scaling to large continually expanding vocabularies under compute-constrained
scenarios. We discover that traditional continual training leads to
catastrophic forgetting under compute constraints, unable to outperform
zero-shot segmentation methods. We introduce a novel strategy for semantic and
panoptic segmentation with zero forgetting, capable of adapting to continually
growing vocabularies without the need for retraining or large memory costs. Our
training-free approach, kNN-CLIP, leverages a database of instance embeddings
to enable open-vocabulary segmentation approaches to continually expand their
vocabulary on any given domain with a single-pass through data, while only
storing embeddings minimizing both compute and memory costs. This method
achieves state-of-the-art mIoU performance across large-vocabulary semantic and
panoptic segmentation datasets. We hope kNN-CLIP represents a step forward in
enabling more efficient and adaptable continual segmentation, paving the way
for advances in real-world large-vocabulary continual segmentation methods.

我们引入了一种新颖的连续语义和全景分割策略，名为 kNN-CLIP，它能够在计算和内存成本最小化的情况下，实现对不断扩大的词汇的适应，并在大词汇语义和全景分割数据集上取得了最先进的性能表现，为实现更高效、可适应性强的连续分割方法迈出了一步。