Open-vocabulary object detection (OVD) has been studied with Vision-Language
Models (VLMs) to detect novel objects beyond the pre-trained categories.
Previous approaches improve the generalization ability to expand the knowledge
of the detector, using 'positive' pseudo-labels with additional 'class' names,
e.g., sock, iPod, and alligator. To extend the previous methods in two aspects,
we propose Retrieval-Augmented Losses and visual Features (RALF). Our method
retrieves related 'negative' classes and augments loss functions. Also, visual
features are augmented with 'verbalized concepts' of classes, e.g., worn on the
feet, handheld music player, and sharp teeth. Specifically, RALF consists of
two modules: Retrieval Augmented Losses (RAL) and Retrieval-Augmented visual
Features (RAF). RAL constitutes two losses reflecting the semantic similarity
with negative vocabularies. In addition, RAF augments visual features with the
verbalized concepts from a large language model (LLM). Our experiments
demonstrate the effectiveness of RALF on COCO and LVIS benchmark datasets. We
achieve improvement up to 3.4 box AP$_{50}^{\text{N}}$ on novel categories of
the COCO dataset and 3.6 mask AP$_{\text{r}}$ gains on the LVIS dataset. Code
is available at this https URL .

使用提取与负词汇的语义相似度和从大型语言模型中取得的概念进行视觉特征增益，通过 Retrieval-Augmented Losses and visual Features (RALF) 方法，成功改进了基于 Vision-Language Models (VLMs) 的开放性词汇目标检测技术，取得了在 COCO 和 LVIS 基准数据集上的显著改进。