Recent years have witnessed remarkable progress in the development of large
vision-language models (LVLMs). Benefiting from the strong language backbones
and efficient cross-modal alignment strategies, LVLMs exhibit surprising
capabilities to perceive visual signals and perform visually grounded
reasoning. However, the capabilities of LVLMs have not been comprehensively and
quantitatively evaluate. Most existing multi-modal benchmarks require
task-oriented input-output formats, posing great challenges to automatically
assess the free-form text output of LVLMs. To effectively leverage the
annotations available in existing benchmarks and reduce the manual effort
required for constructing new benchmarks, we propose to re-formulate existing
benchmarks into unified LVLM-compatible formats. Through systematic data
collection and reformulation, we present the ReForm-Eval benchmark, offering
substantial data for evaluating various capabilities of LVLMs. Based on
ReForm-Eval, we conduct extensive experiments, thoroughly analyze the strengths
and weaknesses of existing LVLMs, and identify the underlying factors. Our
benchmark and evaluation framework will be open-sourced as a cornerstone for
advancing the development of LVLMs.

通过 ReForm-Eval 基准测试，我们对 LVLM 的各种能力进行了全面的定量评估，发现并分析了现有 LVLM 的优点和缺点，并确定了潜在的影响因素。