We propose and study a new computer vision task named open-vocabulary video
instance segmentation (OpenVIS), which aims to simultaneously segment, detect,
and track arbitrary objects in a video according to corresponding text
descriptions. Compared to the original video instance segmentation, OpenVIS
enables users to identify objects of desired categories, regardless of whether
those categories were included in the training dataset. To achieve this goal,
we propose a two-stage pipeline for proposing high-quality class-agnostic
object masks and predicting their corresponding categories via pre-trained VLM.
Specifically, we first employ a query-based mask proposal network to generate
masks of all potential objects, where we replace the original class head with
an instance head trained with a binary object loss, thereby enhancing the
class-agnostic mask proposal ability. Then, we introduce a proposal
post-processing approach to adapt the proposals better to the pre-trained VLMs,
avoiding distortion and unnatural proposal inputs. Meanwhile, to facilitate
research on this new task, we also propose an evaluation benchmark that
utilizes off-the-shelf datasets to comprehensively assess its performance.
Experimentally, the proposed OpenVIS exhibits a remarkable 148\% improvement
compared to the full-supervised baselines on BURST, which have been trained on
all categories.

本研究提出和研究了一项新的计算机视觉任务，名为 OpenVIS，该任务旨在根据对应的文本描述同时分割，检测和跟踪视频中的任意对象。通过开放词汇的设计，OpenVIS 可以识别所需类别的对象，而不管这些类别是否包括在训练数据集中。为了实现这一目标，本文提出了一个由两个阶段组成的流程，首先利用基于查询的蒙版提议网络生成所有潜在对象的蒙版，并通过预先训练的 VLM 预测其对应的类别，其次通过提议后处理方法更好地适应预训练的 VLMs，以避免扭曲和不自然的提议输入。