We propose and study a new computer vision task named open-vocabulary video instance segmentation (OpenVIS), which aims to simultaneously segment, detect, and track arbitrary objects in a video according to corresponding text descriptions. Compared to the original video instance segmentation, OpenVIS enables users to identify objects of desired categories, regardless of whether those categories were included in the training dataset. To achieve this goal, we propose a two-stage pipeline for proposing high-quality class-agnostic object masks and predicting their corresponding categories via pre-trained VLM. Specifically, we first employ a query-based mask proposal network to generate masks of all potential objects, where we replace the original class head with an instance head trained with a binary object loss, thereby enhancing the class-agnostic mask proposal ability. Then, we introduce a proposal post-processing approach to adapt the proposals better to the pre-trained VLMs, avoiding distortion and unnatural proposal inputs. Meanwhile, to facilitate research on this new task, we also propose an evaluation benchmark that utilizes off-the-shelf datasets to comprehensively assess its performance. Experimentally, the proposed OpenVIS exhibits a remarkable 148\% improvement compared to the full-supervised baselines on BURST, which have been trained on all categories.

本研究提出和研究了一项新的计算机视觉任务，名为OpenVIS，该任务旨在根据对应的文本描述同时分割，检测和跟踪视频中的任意对象。通过开放词汇的设计，OpenVIS可以识别所需类别的对象，而不管这些类别是否包括在训练数据集中。为了实现这一目标，本文提出了一个由两个阶段组成的流程，首先利用基于查询的蒙版提议网络生成所有潜在对象的蒙版，并通过预先训练的VLM预测其对应的类别，其次通过提议后处理方法更好地适应预训练的VLMs，以避免扭曲和不自然的提议输入。

OpenVIS: 开放词汇视频实例分割