Existing open-vocabulary object detectors typically enlarge their vocabulary
sizes by leveraging different forms of weak supervision. This helps generalize
to novel objects at inference. Two popular forms of weak-supervision used in
open-vocabulary detection (OVD) include pretrained CLIP model and image-level
supervision. We note that both these modes of supervision are not optimally
aligned for the detection task: CLIP is trained with image-text pairs and lacks
precise localization of objects while the image-level supervision has been used
with heuristics that do not accurately specify local object regions. In this
work, we propose to address this problem by performing object-centric alignment
of the language embeddings from the CLIP model. Furthermore, we visually ground
the objects with only image-level supervision using a pseudo-labeling process
that provides high-quality object proposals and helps expand the vocabulary
during training. We establish a bridge between the above two object-alignment
strategies via a novel weight transfer function that aggregates their
complimentary strengths. In essence, the proposed model seeks to minimize the
gap between object and image-centric representations in the OVD setting. On the
COCO benchmark, our proposed approach achieves 36.6 AP50 on novel classes, an
absolute 8.2 gain over the previous best performance. For LVIS, we surpass the
state-of-the-art ViLD model by 5.0 mask AP for rare categories and 3.4 overall.
Code: this https URL.

本文提出了一种基于 CLIP 模型和图像水平监督的对象中心对齐的方法，运用伪标注实现高质量对象提议并在训练过程中扩展词汇表，通过新的权重传递函数将两种对象对齐策略结合，实现了在 OVD 方案中对象和图像中心表示的最小化差距。在 COCO 数据集上，我们的方法在新颖类别上取得了 36.6 的 AP50 表现，绝对值超过了以前的最佳性能。对于 LVIS，我们在罕见类别上超越了最新的 ViLD 模型达 5.0 的掩膜 AP，总体提高 3.4。