Visual grounding is an essential tool that links user-provided text queries
with query-specific regions within an image. Despite advancements in visual
grounding models, their ability to comprehend complex queries remains limited.
To overcome this limitation, we introduce LLM-Optic, an innovative method that
utilizes Large Language Models (LLMs) as an optical lens to enhance existing
visual grounding models in comprehending complex text queries involving
intricate text structures, multiple objects, or object spatial relationships,
situations that current models struggle with. LLM-Optic first employs an LLM as
a Text Grounder to interpret complex text queries and accurately identify
objects the user intends to locate. Then a pre-trained visual grounding model
is used to generate candidate bounding boxes given the refined query by the
Text Grounder. After that, LLM-Optic annotates the candidate bounding boxes
with numerical marks to establish a connection between text and specific image
regions, thereby linking two distinct modalities. Finally, it employs a Large
Multimodal Model (LMM) as a Visual Grounder to select the marked candidate
objects that best correspond to the original text query. Through LLM-Optic, we
have achieved universal visual grounding, which allows for the detection of
arbitrary objects specified by arbitrary human language input. Importantly, our
method achieves this enhancement without requiring additional training or
fine-tuning. Extensive experiments across various challenging benchmarks
demonstrate that LLM-Optic achieves state-of-the-art zero-shot visual grounding
capabilities. Project Page: this https URL

通过使用大型语言模型作为视觉定位模型的辅助工具，LLM-Optic 方法克服了复杂文本查询的限制，实现了可以通过任意语言输入来检测任意对象的视觉定位能力，无需额外训练或微调。