Vision Language Models (VLMs) demonstrate remarkable proficiency in
addressing a wide array of visual questions, which requires strong perception
and reasoning faculties. Assessing these two competencies independently is
crucial for model refinement, despite the inherent difficulty due to the
intertwined nature of seeing and reasoning in existing VLMs. To tackle this
issue, we present Prism, an innovative framework designed to disentangle the
perception and reasoning processes involved in visual question solving. Prism
comprises two distinct stages: a perception stage that utilizes a VLM to
extract and articulate visual information in textual form, and a reasoning
stage that formulates responses based on the extracted visual information using
a Large Language Model (LLM). This modular design enables the systematic
comparison and assessment of both proprietary and open-source VLM for their
perception and reasoning strengths. Our analytical framework provides several
valuable insights, underscoring Prism's potential as a cost-effective solution
for vision-language tasks. By combining a streamlined VLM focused on perception
with a powerful LLM tailored for reasoning, Prism achieves superior results in
general vision-language tasks while substantially cutting down on training and
operational expenses. Quantitative evaluations show that Prism, when configured
with a vanilla 2B LLaVA and freely accessible GPT-3.5, delivers performance on
par with VLMs $10 \times$ larger on the rigorous multimodal benchmark MMStar.
The project is released at: this https URL

Prism 是一个创新的框架，旨在解决视觉问题求解中构成的知觉和推理过程的相互交织难题，通过将感知和推理分离为两个不同的阶段，实现了对专有和开源 Vision Language Models 在感知和推理能力上的系统比较和评估。通过将专注于感知的精简 VLM 与专为推理而设计的强大 LLM 相结合，Prism 在一般的视觉语言任务中取得了卓越的结果，同时大幅减少了训练和操作成本的投入。在严格的多模式基准 MMStar 上，使用 vanilla 2B LLaVA 和免费获取的 GPT-3.5 配置 Prism，在性能上媲美比其大 10 倍的 VLM。