Recent advancements in generative AI have suggested that by taking visual
prompt, GPT-4V can demonstrate significant proficiency in image recognition
task. Despite its impressive capabilities, the financial cost associated with
GPT-4V's inference presents a substantial barrier for its wide use. To address
this challenge, our work introduces Collage Prompting, a budget-friendly
prompting approach that concatenates multiple images into a single visual
input. With collage prompt, GPT-4V is able to perform image recognition on
several images simultaneously. Based on the observation that the accuracy of
GPT-4V's image recognition varies significantly with the order of images within
the collage prompt, our method further learns to optimize the arrangement of
images for maximum recognition accuracy. A graph predictor is trained to
indicate the accuracy of each collage prompt, then we propose an optimization
method to navigate the search space of possible image arrangements. Experiment
results across various datasets demonstrate the cost-efficiency score of
collage prompt is much larger than standard prompt. Additionally, collage
prompt with learned arrangement achieves clearly better accuracy than collage
prompt with random arrangement in GPT-4V's visual recognition.

我们的研究引入了拼贴提示，一种预算友好的提示方法，在 GPT-4V 上实现了多张图像的同时图像识别，并通过优化图像排列以达到最大的识别准确性，实验证明拼贴提示在成本效益上明显优于标准提示，并且学习的排列方式明显比随机排列在 GPT-4V 的视觉识别中更准确。