Foundation models, pre-trained on a large amount of data have demonstrated
impressive zero-shot capabilities in various downstream tasks. However, in
object detection and instance segmentation, two fundamental computer vision
tasks heavily reliant on extensive human annotations, foundation models such as
SAM and DINO struggle to achieve satisfactory performance. In this study, we
reveal that the devil is in the object boundary, \textit{i.e.}, these
foundation models fail to discern boundaries between individual objects. For
the first time, we probe that CLIP, which has never accessed any instance-level
annotations, can provide a highly beneficial and strong instance-level boundary
prior in the clustering results of its particular intermediate layer. Following
this surprising observation, we propose $\textbf{Zip}$ which $\textbf{Z}$ips up
CL$\textbf{ip}$ and SAM in a novel classification-first-then-discovery
pipeline, enabling annotation-free, complex-scene-capable, open-vocabulary
object detection and instance segmentation. Our Zip significantly boosts SAM's
mask AP on COCO dataset by 12.5% and establishes state-of-the-art performance
in various settings, including training-free, self-training, and
label-efficient finetuning. Furthermore, annotation-free Zip even achieves
comparable performance to the best-performing open-vocabulary object detecters
using base annotations. Code is released at
this https URL

基于大量数据预训练的模型在各种下游任务中展示出令人印象深刻的零摸索能力，但是在目标检测和实例分割等基本计算机视觉任务中，这些基础模型（如 SAM 和 DINO）难以达到令人满意的性能。本研究揭示了根源在于目标边界，即这些基础模型无法区分个体对象之间的边界。我们首次证明 CLIP，它从未访问过任何实例级注释，可以在其特定中间层的聚类结果中提供极具益处和强大的实例级边界先验。基于这一令人惊讶的观察，我们提出了 Zip，它在一个新颖的 “先分类，然后发现” 管道中将 CLip 和 SAM 结合起来，从而实现了无需注释、适应复杂场景并具有开放词汇的目标检测和实例分割。我们的 Zip 显著提升了在 COCO 数据集上 SAM 的遮罩 AP 值达到了 12.5%，并在各种设置中建立了最先进的性能，包括无需训练、自训练和标签高效微调。此外，无需注释的 Zip 甚至实现了与使用基本注释的最佳开放词汇目标检测器相媲美的性能。代码在此 https URL 上发布。