Vision Language Models (VLMs) demonstrate remarkable proficiency in
addressing a wide array of visual questions, which requires strong perception
and reasoning faculties. Assessing these two competencies independently is
crucial for model refinement, despite the inherent difficulty due to the
intertwined nature of seeing and reasoning in existing VLMs. To tackle this
issue, we present Prism, an innovative framework designed to disentangle the
perception and reasoning processes involved in visual question solving. Prism
comprises two distinct stages: a perception stage that utilizes a VLM to
extract and articulate visual information in textual form, and a reasoning
stage that formulates responses based on the extracted visual information using
a Large Language Model (LLM). This modular design enables the systematic
comparison and assessment of both proprietary and open-source VLM for their
perception and reasoning strengths. Our analytical framework provides several
valuable insights, underscoring Prism's potential as a cost-effective solution
for vision-language tasks. By combining a streamlined VLM focused on perception
with a powerful LLM tailored for reasoning, Prism achieves superior results in
general vision-language tasks while substantially cutting down on training and
operational expenses. Quantitative evaluations show that Prism, when configured
with a vanilla 2B LLaVA and freely accessible GPT-3.5, delivers performance on
par with VLMs $10 \times$ larger on the rigorous multimodal benchmark MMStar.
The project is released at: this https URL

Prism 是一个创新的框架，旨在解决视觉问题求解中构成的知觉和推理过程的相互交织难题，通过将感知和推理分离为两个不同的阶段，实现了对专有和开源 Vision Language Models 在感知和推理能力上的系统比较和评估。通过将专注于感知的精简 VLM 与专为推理而设计的强大 LLM 相结合，Prism 在一般的视觉语言任务中取得了卓越的结果，同时大幅减少了训练和操作成本的投入。在严格的多模式基准 MMStar 上，使用 vanilla 2B LLaVA 和免费获取的 GPT-3.5 配置 Prism，在性能上媲美比其大 10 倍的 VLM。

Prism: 一个解耦和评估 VLM 能力的框架

Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs

The rapid advancement of open-source foundation models has brought
transparency and accessibility to this groundbreaking technology. However, this
openness has also enabled the development of highly-capable, unsafe models, as
exemplified by recent instances such as WormGPT and FraudGPT, which are
specifically designed to facilitate criminal activity. As the capabilities of
open foundation models continue to grow, potentially outpacing those of
closed-source models, the risk of misuse by bad actors poses an increasingly
serious threat to society. This paper addresses the critical question of how
open foundation model developers should approach model safety in light of these
challenges. Our analysis reveals that open-source foundation model companies
often provide less restrictive acceptable use policies (AUPs) compared to their
closed-source counterparts, likely due to the inherent difficulties in
enforcing such policies once the models are released. To tackle this issue, we
introduce PRISM, a design framework for open-source foundation model safety
that emphasizes Private, Robust, Independent Safety measures, at Minimal
marginal cost of compute. The PRISM framework proposes the use of modular
functions that moderate prompts and outputs independently of the core language
model, offering a more adaptable and resilient approach to safety compared to
the brittle reinforcement learning methods currently used for value alignment.
By focusing on identifying AUP violations and engaging the developer community
in establishing consensus around safety design decisions, PRISM aims to create
a safer open-source ecosystem that maximizes the potential of these powerful
technologies while minimizing the risks to individuals and society as a whole.

在开源基础模型的快速发展中，模型安全性问题变得尤为重要。本研究提出了 PRISM 框架，该框架强调私密性、鲁棒性、独立性和计算成本最小化，并通过识别使用政策违规行为，与开发者共同建立安全设计决策的共识，为创建最大化技术潜力并最小化个人和整个社会风险的安全开源生态系统做出贡献。