Task driven object detection aims to detect object instances suitable for
affording a task in an image. Its challenge lies in object categories available
for the task being too diverse to be limited to a closed set of object
vocabulary for traditional object detection. Simply mapping categories and
visual features of common objects to the task cannot address the challenge. In
this paper, we propose to explore fundamental affordances rather than object
categories, i.e., common attributes that enable different objects to accomplish
the same task. Moreover, we propose a novel multi-level chain-of-thought
prompting (MLCoT) to extract the affordance knowledge from large language
models, which contains multi-level reasoning steps from task to object examples
to essential visual attributes with rationales. Furthermore, to fully exploit
knowledge to benefit object recognition and localization, we propose a
knowledge-conditional detection framework, namely CoTDet. It conditions the
detector from the knowledge to generate object queries and regress boxes.
Experimental results demonstrate that our CoTDet outperforms state-of-the-art
methods consistently and significantly (+15.6 box AP and +14.8 mask AP) and can
generate rationales for why objects are detected to afford the task.

本文提出了基于任务驱动的物体检测方法，通过探索基本意义而非物体类别来提取关键属性，并使用多级思维链激励法从大规模语言模型中提取与任务、物体示例和关键视觉属性相关的知识。进一步，我们提出了一种知识条件检测框架 CoTDet，有效利用知识来增强物体识别和定位，并得到支持物体检测的合理解释。实验证明，我们的 CoTDet 方法在性能上显著优于当前最先进的方法（提升 15.6 个方框 AP 值和 14.8 个掩膜 AP 值）。