Recent studies show promising performance in open-vocabulary object detection
(OVD) using pseudo labels (PLs) from pretrained vision and language models
(VLMs). However, PLs generated by VLMs are extremely noisy due to the gap
between the pretraining objective of VLMs and OVD, which blocks further
advances on PLs. In this paper, we aim to reduce the noise in PLs and propose a
method called online Self-training And a Split-and-fusion head for OVD
(SAS-Det). First, the self-training finetunes VLMs to generate high quality PLs
while prevents forgetting the knowledge learned in the pretraining. Second, a
split-and-fusion (SAF) head is designed to remove the noise in localization of
PLs, which is usually ignored in existing methods. It also fuses complementary
knowledge learned from both precise ground truth and noisy pseudo labels to
boost the performance. Extensive experiments demonstrate SAS-Det is both
efficient and effective. Our pseudo labeling is 3 times faster than prior
methods. SAS-Det outperforms prior state-of-the-art models of the same scale by
a clear margin and achieves 37.4 AP$_{50}$ and 27.3 AP$_r$ on novel categories
of the COCO and LVIS benchmarks, respectively.

通过在线自训练和分割融合头以减少预训练视觉语言模型生成的噪声伪标签并提高开放词汇目标检测性能。