We present a new open-vocabulary detection approach based on
detection-oriented image-text pretraining to bridge the gap between image-level
pretraining and open-vocabulary object detection. At the pretraining phase, we
replace the commonly used classification architecture with the detector
architecture, which better serves the region-level recognition needs of
detection by enabling the detector heads to learn from noisy image-text pairs.
Using only standard contrastive loss and no pseudo-labeling, our approach is a
simple yet effective extension of the contrastive learning method to learn
emergent object-semantic cues. In addition, we propose a shifted-window
learning approach upon window attention to make the backbone representation
more robust, translation-invariant, and less biased by the window pattern. On
the popular LVIS open-vocabulary detection benchmark, our approach sets a new
state of the art of 40.4 mask AP$_r$ using the common ViT-L backbone,
significantly outperforming the best existing approach by +6.5 mask AP$_r$ at
system level. On the COCO benchmark, we achieve very competitive 40.8 novel AP
without pseudo labeling or weak supervision. In addition, we evaluate our
approach on the transfer detection setup, where ours outperforms the baseline
significantly. Visualization reveals emerging object locality from the
pretraining recipes compared to the baseline. Code and models will be publicly
released.

基于检测导向的图像 - 文本预训练的新的开放词汇检测方法用于填补图像级预训练和开放词汇对象检测之间的差距，通过使探测器头从嘈杂的图像 - 文本对中学习，我们的方法能够利用对比损失学习到新出现的对象 - 语义线索，在 LVIS 和 COCO 基准测试中均获得了非常有竞争力的结果，并在转移检测设置中显著优于基线。