Evaluating the alignment capabilities of large Vision-Language Models (VLMs)
is essential for determining their effectiveness as helpful assistants.
However, existing benchmarks primarily focus on basic abilities using nonverbal
methods, such as yes-no and multiple-choice questions. In this paper, we
address this gap by introducing AlignMMBench, a comprehensive alignment
benchmark specifically designed for emerging Chinese VLMs. This benchmark is
meticulously curated from real-world scenarios and Chinese Internet sources,
encompassing thirteen specific tasks across three categories, and includes both
single-turn and multi-turn dialogue scenarios. Incorporating a prompt rewrite
strategy, AlignMMBench encompasses 1,054 images and 4,978 question-answer
pairs. To facilitate the evaluation pipeline, we propose CritiqueVLM, a
rule-calibrated evaluator that exceeds GPT-4's evaluation ability. Finally, we
report the performance of representative VLMs on AlignMMBench, offering
insights into the capabilities and limitations of different VLM architectures.
All evaluation codes and data are available on this https URL

本研究通过引入 AlignMMBench，一个专门为新兴的中文视觉 - 语言模型设计的综合对齐基准，从真实场景和中国互联网来源精心策划，并包括三个类别中的十三个具体任务，以及单轮和多轮对话场景。通过结合一个提示重写策略，AlignMMBench 包括 1054 个图像和 4978 个问答对。为了促进评估流程，我们提出了 CritiqueVLM，一个超越 GPT-4 评估能力的规则校准评估器。最后，我们报告了 AlignMMBench 上代表性 VLM 的性能，提供了不同 VLM 架构的能力和限制的见解。