Existing open-vocabulary object detectors typically require a predefined set
of categories from users, significantly confining their application scenarios.
In this paper, we introduce DetCLIPv3, a high-performing detector that excels
not only at both open-vocabulary object detection, but also generating
hierarchical labels for detected objects. DetCLIPv3 is characterized by three
core designs: 1. Versatile model architecture: we derive a robust open-set
detection framework which is further empowered with generation ability via the
integration of a caption head. 2. High information density data: we develop an
auto-annotation pipeline leveraging visual large language model to refine
captions for large-scale image-text pairs, providing rich, multi-granular
object labels to enhance the training. 3. Efficient training strategy: we
employ a pre-training stage with low-resolution inputs that enables the object
captioner to efficiently learn a broad spectrum of visual concepts from
extensive image-text paired data. This is followed by a fine-tuning stage that
leverages a small number of high-resolution samples to further enhance
detection performance. With these effective designs, DetCLIPv3 demonstrates
superior open-vocabulary detection performance, \eg, our Swin-T backbone model
achieves a notable 47.0 zero-shot fixed AP on the LVIS minival benchmark,
outperforming GLIPv2, GroundingDINO, and DetCLIPv2 by 18.0/19.6/6.6 AP,
respectively. DetCLIPv3 also achieves a state-of-the-art 19.7 AP in dense
captioning task on VG dataset, showcasing its strong generative capability.

DetCLIPv3 是一种高性能检测器，不仅在开放词汇物体检测方面表现出色，还能为检测到的物体生成分层标签。