We present DetToolChain, a novel prompting paradigm, to unleash the zero-shot
object detection ability of multimodal large language models (MLLMs), such as
GPT-4V and Gemini. Our approach consists of a detection prompting toolkit
inspired by high-precision detection priors and a new Chain-of-Thought to
implement these prompts. Specifically, the prompts in the toolkit are designed
to guide the MLLM to focus on regional information (e.g., zooming in), read
coordinates according to measure standards (e.g., overlaying rulers and
compasses), and infer from the contextual information (e.g., overlaying scene
graphs). Building upon these tools, the new detection chain-of-thought can
automatically decompose the task into simple subtasks, diagnose the
predictions, and plan for progressive box refinements. The effectiveness of our
framework is demonstrated across a spectrum of detection tasks, especially hard
cases. Compared to existing state-of-the-art methods, GPT-4V with our
DetToolChain improves state-of-the-art object detectors by +21.5% AP50 on MS
COCO Novel class set for open-vocabulary detection, +24.23% Acc on RefCOCO val
set for zero-shot referring expression comprehension, +14.5% AP on D-cube
describe object detection FULL setting.

DetToolChain 是一种新颖的提示范式，通过高精度检测先验和一种新的思维链条，实现多模态大语言模型（如 GPT-4V 和 Gemini）的零样本目标检测能力。