This paper presents a new framework for open-vocabulary semantic segmentation
with the pre-trained vision-language model, named Side Adapter Network (SAN).
Our approach models the semantic segmentation task as a region recognition
problem. A side network is attached to a frozen CLIP model with two branches:
one for predicting mask proposals, and the other for predicting attention bias
which is applied in the CLIP model to recognize the class of masks. This
decoupled design has the benefit CLIP in recognizing the class of mask
proposals. Since the attached side network can reuse CLIP features, it can be
very light. In addition, the entire network can be trained end-to-end, allowing
the side network to be adapted to the frozen CLIP model, which makes the
predicted mask proposals CLIP-aware. Our approach is fast, accurate, and only
adds a few additional trainable parameters. We evaluate our approach on
multiple semantic segmentation benchmarks. Our method significantly outperforms
other counterparts, with up to 18 times fewer trainable parameters and 19 times
faster inference speed. We hope our approach will serve as a solid baseline and
help ease future research in open-vocabulary semantic segmentation. The code
will be available at this https URL.

本文介绍了一种利用预训练的视觉 - 语言模型（名为 Side Adapter Network）实现开放词汇语义分割的新框架，其中包括对固定 CLIP 模型附加侧网络以识别类别和预测掩模建议等过程，最终实现快速而精确的语义分割，并在多个基准测试集上超越了其他竞争者，还拥有较少的可训练参数和较快的推断速度。