Task-oriented object detection aims to find objects suitable for
accomplishing specific tasks. As a challenging task, it requires simultaneous
visual data processing and reasoning under ambiguous semantics. Recent
solutions are mainly all-in-one models. However, the object detection backbones
are pre-trained without text supervision. Thus, to incorporate task
requirements, their intricate models undergo extensive learning on a highly
imbalanced and scarce dataset, resulting in capped performance, laborious
training, and poor generalizability. In contrast, we propose TaskCLIP, a more
natural two-stage design composed of general object detection and task-guided
object selection. Particularly for the latter, we resort to the recently
successful large Vision-Language Models (VLMs) as our backbone, which provides
rich semantic knowledge and a uniform embedding space for images and texts.
Nevertheless, the naive application of VLMs leads to sub-optimal quality, due
to the misalignment between embeddings of object images and their visual
attributes, which are mainly adjective phrases. To this end, we design a
transformer-based aligner after the pre-trained VLMs to re-calibrate both
embeddings. Finally, we employ a trainable score function to post-process the
VLM matching results for object selection. Experimental results demonstrate
that our TaskCLIP outperforms the state-of-the-art DETR-based model TOIST by
3.5% and only requires a single NVIDIA RTX 4090 for both training and
inference.

通过使用大规模视觉和语言模型（VLM）作为模型骨干，以及基于 Transformer 的校准器，我们提出了 TaskCLIP，一个两阶段的设计用于任务导向的目标检测，实验结果表明我们的方法优于当前最先进的 TOIST 模型，并且只需要一张 NVIDIA RTX 4090 显卡进行训练和推理。