Advancing object detection to open-vocabulary and few-shot transfer has long
been a challenge for computer vision research. This work explores a continual
learning approach that enables a detector to expand its zero/few-shot
capabilities via multi-dataset vision-language pre-training. Using natural
language as knowledge representation, we explore methods to accumulate "visual
vocabulary" from different training datasets and unify the task as a
language-conditioned detection framework. Specifically, we propose a novel
language-aware detector OmDet and a novel training mechanism. The proposed
multimodal detection network can resolve the technical challenges in
multi-dataset joint training and it can generalize to arbitrary number of
training datasets without the requirements for manual label taxonomy merging.
Experiment results on COCO, Pascal VOC, and Wider Face/Pedestrian confirmed the
efficacy by achieving on par or higher scores in joint training compared to
training separately. Moreover, we pre-train on more than 20 million images with
4 million unique object vocabulary, and the resulting model is evaluated on 35
downstream tasks of ODinW. Results show that OmDet is able to achieve the
state-of-the-art fine-tuned performance on ODinW. And analysis shows that by
scaling up the proposed pre-training method, OmDet continues to improve its
zero/few-shot tuning performance, suggesting a promising way for further
scaling.

本文提出一种基于多模态预训练的语言条件检测框架，能够通过自然语言作为知识表示从不同的数据集中积累 “视觉词汇”，解决多数据集训练中的技术难题，并成功应用于超过 35 个目标检测任务，取得最新成果。