In this paper, we delve into the realm of vision transformers for continual
semantic segmentation, a problem that has not been sufficiently explored in
previous literature. Empirical investigations on the adaptation of existing
frameworks to vanilla ViT reveal that incorporating visual adapters into ViTs
or fine-tuning ViTs with distillation terms is advantageous for enhancing the
segmentation capability of novel classes. These findings motivate us to propose
Continual semantic Segmentation via Adapter-based ViT, namely ConSept. Within
the simplified architecture of ViT with linear segmentation head, ConSept
integrates lightweight attention-based adapters into vanilla ViTs. Capitalizing
on the feature adaptation abilities of these adapters, ConSept not only retains
superior segmentation ability for old classes, but also attains promising
segmentation quality for novel classes. To further harness the intrinsic
anti-catastrophic forgetting ability of ConSept and concurrently enhance the
segmentation capabilities for both old and new classes, we propose two key
strategies: distillation with a deterministic old-classes boundary for improved
anti-catastrophic forgetting, and dual dice losses to regularize segmentation
maps, thereby improving overall segmentation performance. Extensive experiments
show the effectiveness of ConSept on multiple continual semantic segmentation
benchmarks under overlapped or disjoint settings. Code will be publicly
available at https://github.com/DongSky/ConSept.

通过向 vanilla ViTs 集成轻量级的 adapter，提出了 ConSept 用于不断的语义分割，在旧类别和新类别的分割能力上取得了显著提升，并通过确定性的旧类别边界进行蒸馏以提高抗灾性遗忘能力并改善整体分割性能。

ConSept：基于适配器的视觉 Transformer 的持续语义分割

ConSept: Continual Semantic Segmentation via Adapter-based Vision  Transformer

Existing open-vocabulary image segmentation methods require a fine-tuning
step on mask annotations and/or image-text datasets. Mask labels are
labor-intensive, which limits the number of categories in segmentation
datasets. As a result, the open-vocabulary capacity of pre-trained VLMs is
severely reduced after fine-tuning. However, without fine-tuning, VLMs trained
under weak image-text supervision tend to make suboptimal mask predictions when
there are text queries referring to non-existing concepts in the image. To
alleviate these issues, we introduce a novel recurrent framework that
progressively filters out irrelevant texts and enhances mask quality without
training efforts. The recurrent unit is a two-stage segmenter built upon a VLM
with frozen weights. Thus, our model retains the VLM's broad vocabulary space
and strengthens its segmentation capability. Experimental results show that our
method outperforms not only the training-free counterparts, but also those
fine-tuned with millions of additional data samples, and sets new
state-of-the-art records for both zero-shot semantic and referring image
segmentation tasks. Specifically, we improve the current record by 28.8, 16.0,
and 6.9 mIoU on Pascal VOC, COCO Object, and Pascal Context.

通过引入一个创新的递归框架，我们的研究表明，在不进行训练的情况下，我们的模型能够优于那些经过百万级附加样本微调的方法，为零样本语义和参考图像分割任务的最新记录设定了新的技术水平。

CLIP 作为 RNN：无需培训即可分割无限的视觉概念

CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor

Although perception systems have made remarkable advancements in recent
years, they still rely on explicit human instruction to identify the target
objects or categories before executing visual recognition tasks. Such systems
lack the ability to actively reason and comprehend implicit user intentions. In
this work, we propose a new segmentation task -- reasoning segmentation. The
task is designed to output a segmentation mask given a complex and implicit
query text. Furthermore, we establish a benchmark comprising over one thousand
image-instruction pairs, incorporating intricate reasoning and world knowledge
for evaluation purposes. Finally, we present LISA: large Language Instructed
Segmentation Assistant, which inherits the language generation capabilities of
the multi-modal Large Language Model (LLM) while also possessing the ability to
produce segmentation masks. We expand the original vocabulary with a <SEG>
token and propose the embedding-as-mask paradigm to unlock the segmentation
capability. Remarkably, LISA can handle cases involving: 1) complex reasoning;
2) world knowledge; 3) explanatory answers; 4) multi-turn conversation. Also,
it demonstrates robust zero-shot capability when trained exclusively on
reasoning-free datasets. In addition, fine-tuning the model with merely 239
reasoning segmentation image-instruction pairs results in further performance
enhancement. Experiments show our method not only unlocks new reasoning
segmentation capabilities but also proves effective in both complex reasoning
segmentation and standard referring segmentation tasks. Code, models, and demo
are at this https URL

该研究提出了一种新的推理分割任务，以激活感知系统中的推理分割能力，并展示了多模态语言模型 LISA 在复杂推理分割和标准引用分割任务中的有效性。