Multimodal Large Language Models (MLLMs) such as GPT-4V and Gemini Pro face
challenges in achieving human-level perception in Visual Question Answering
(VQA), particularly in object-oriented perception tasks which demand
fine-grained understanding of object identities, locations or attributes, as
indicated by empirical findings. This is mainly due to their limited capability
to effectively integrate complex visual cues with textual information and
potential object hallucinations. In this paper, we present a novel approach,
Joint Visual and Text Prompting (VTPrompt), that employs fine-grained visual
information to enhance the capability of MLLMs in VQA, especially for
object-oriented perception. VTPrompt merges visual and text prompts to extract
key concepts from textual questions and employs a detection model to highlight
relevant objects as visual prompts in images. The processed images alongside
text prompts are subsequently fed into MLLMs to produce more accurate answers.
Our experiments with GPT-4V and Gemini Pro, on three benchmarks, i.e., MME ,
MMB and POPE, demonstrate significant improvements. Particularly, our method
led to a score improvement of up to 183.5 for GPT-4V on MME and enhanced MMB
performance by 8.17\% for GPT-4V and 15.69\% for Gemini Pro.

使用视觉和文本提示的新方法 (VTPrompt) 提高了 Multimodal Large Language Models 在物体感知方面的能力，并在三个基准测试上表现出显著的改进。