In recent times, following the paradigm of DETR (DEtection TRansformer),
query-based end-to-end instance segmentation (QEIS) methods have exhibited
superior performance compared to CNN-based models, particularly when trained on
large-scale datasets. Nevertheless, the effectiveness of these QEIS methods
diminishes significantly when confronted with limited training data. This
limitation arises from their reliance on substantial data volumes to
effectively train the pivotal queries/kernels that are essential for acquiring
localization and shape priors. To address this problem, we propose a novel
method for unsupervised pre-training in low-data regimes. Inspired by the
recently successful prompting technique, we introduce a new method,
Unsupervised Pre-training with Language-Vision Prompts (UPLVP), which improves
QEIS models' instance segmentation by bringing language-vision prompts to
queries/kernels. Our method consists of three parts: (1) Masks Proposal:
Utilizes language-vision models to generate pseudo masks based on unlabeled
images. (2) Prompt-Kernel Matching: Converts pseudo masks into prompts and
injects the best-matched localization and shape features to their corresponding
kernels. (3) Kernel Supervision: Formulates supervision for pre-training at the
kernel level to ensure robust learning. With the help of our pre-training
method, QEIS models can converge faster and perform better than CNN-based
models in low-data regimes. Experimental evaluations conducted on MS COCO,
Cityscapes, and CTW1500 datasets indicate that the QEIS models' performance can
be significantly improved when pre-trained with our method. Code will be
available at: this https URL

在当前的研究中，根据最新的 DETR（DEtection TRansformer）范式，基于查询的端到端实例分割（QEIS）方法在大规模数据集上训练时表现出了卓越的性能，尤其是与基于 CNN 的模型相比。然而，当面对有限的训练数据时，这些 QEIS 方法的有效性显著降低。为解决这一问题，我们提出了一种新颖的方法，用于无监督预训练在低数据环境中，即 Unsupervised Pre-training with Language-Vision Prompts (UPLVP)，通过将语言 - 视觉提示引入查询 / 核心，改善了 QEIS 模型的实例分割。我们的方法包括三个部分：(1) Mask Proposal：利用语言 - 视觉模型基于无标签图像生成伪掩膜。(2) Prompt-Kernel Matching：将伪掩膜转换为提示，并将最佳匹配的定位和形状特征注入到对应的核心中。(3) Kernel Supervision：以核心级别制定预训练的监督，以确保鲁棒学习。在我们的预训练方法的帮助下，QEIS 模型可以在低数据环境中更快地收敛并表现出更好的性能，实验评估表明，当使用我们的方法进行预训练时，QEIS 模型在 MS COCO、Cityscapes 和 CTW1500 数据集上的表现可以显著提高。代码将在此 https URL 中提供。