Open-vocabulary video instance segmentation strives to segment and track
instances belonging to an open set of categories in a video. The
vision-language model Contrastive Language-Image Pre-training (CLIP) has shown
strong zero-shot classification ability in image-level open-vocabulary task. In
this paper, we propose a simple encoder-decoder network, called CLIP-VIS, to
adapt CLIP for open-vocabulary video instance segmentation. Our CLIP-VIS adopts
frozen CLIP image encoder and introduces three modules, including
class-agnostic mask generation, temporal topK-enhanced matching, and weighted
open-vocabulary classification. Given a set of initial queries, class-agnostic
mask generation employs a transformer decoder to predict query masks and
corresponding object scores and mask IoU scores. Then, temporal topK-enhanced
matching performs query matching across frames by using K mostly matched
frames. Finally, weighted open-vocabulary classification first generates query
visual features with mask pooling, and second performs weighted classification
using object scores and mask IoU scores. Our CLIP-VIS does not require the
annotations of instance categories and identities. The experiments are
performed on various video instance segmentation datasets, which demonstrate
the effectiveness of our proposed method, especially on novel categories. When
using ConvNeXt-B as backbone, our CLIP-VIS achieves the AP and APn scores of
32.1% and 40.3% on validation set of LV-VIS dataset, which outperforms OV2Seg
by 11.0% and 24.0% respectively. We will release the source code and models at
this https URL

本研究提出了一个名为 CLIP-VIS 的简单编码器 - 解码器网络，用于自适应开放词汇视频实例分割。CLIP-VIS 采用冻结的 CLIP 图像编码器，并引入了类不可知的掩码生成、时序 Top-K 增强匹配和加权开放词汇分类三个模块，实验结果表明该方法在各种视频实例分割数据集上表现出色。