In the pursuit of Artificial General Intelligence (AGI), the integration of
vision in language models has marked a significant milestone. The advent of
vision-language models (MLLMs) like GPT-4V have expanded AI applications,
aligning with the multi-modal capabilities of the human brain. However,
evaluating the efficacy of MLLMs poses a substantial challenge due to the
subjective nature of tasks that lack definitive answers. Existing automatic
evaluation methodologies on multi-modal large language models rely on objective
queries that have standard answers, inadequately addressing the nuances of
creative and associative multi-modal tasks. To address this, we introduce
MLLM-Bench, an innovative benchmark inspired by Vicuna, spanning a diverse
array of scenarios, including Perception, Understanding, Applying, Analyzing,
Evaluating, and Creation along with the ethical consideration. MLLM-Bench is
designed to reflect user experience more accurately and provide a more holistic
assessment of model performance. Comparative evaluations indicate a significant
performance gap between existing open-source models and GPT-4V. We posit that
MLLM-Bench will catalyze progress in the open-source community towards
developing user-centric vision-language models that meet a broad spectrum of
real-world applications. See online leaderboard in
https://mllm-bench.llmzoo.com.

为了追求人工通用智能（AGI），将视觉集成到语言模型中标志着一个重要里程碑。视觉语言模型（MLLMs）的出现，如 GPT-4V，扩展了人工智能应用程序，与人脑的多模态能力相匹配。然而，评估 MLLMs 的有效性面临着重大挑战，因为缺乏确定性答案的任务具有主观性。现有的多模态大型语言模型的自动评估方法依赖于具有标准答案的客观查询，并未充分解决创造性和联想性多模态任务的细微差别。为了解决这个问题，我们引入了 MLLM-Bench，这是一个受 Vicuna 启发的创新基准，涵盖了各种场景，包括感知、理解、应用、分析、评估和创作，以及伦理考虑。MLLM-Bench 的设计更加准确地反映用户体验，并提供了对模型性能更全面的评估。对比评估结果表明，现有的开源模型和 GPT-4V 之间存在显著的性能差距。我们认为，MLLM-Bench 将推动开源社区在开发能满足广泛实际应用需求的用户导向视觉语言模型方面取得进展。请访问 https://mllm-bench.llmzoo.com 查看在线排行榜。

MLLM-Bench，使用 GPT-4V 评估多模式 LLMs

MLLM-Bench, Evaluating Multi-modal LLMs using GPT-4V

As LLMs have become increasingly popular, they have been used in almost every
field. But as the application for LLMs expands from generic fields to narrow,
focused science domains, there exists an ever-increasing gap in ways to
evaluate their efficacy in those fields. For the benchmarks that do exist, a
lot of them focus on questions that don't require proper understanding of the
subject in question. In this paper, we present NuclearQA, a human-made
benchmark of 100 questions to evaluate language models in the nuclear domain,
consisting of a varying collection of questions that have been specifically
designed by experts to test the abilities of language models. We detail our
approach and show how the mix of several types of questions makes our benchmark
uniquely capable of evaluating models in the nuclear domain. We also present
our own evaluation metric for assessing LLM's performances due to the
limitations of existing ones. Our experiments on state-of-the-art models
suggest that even the best LLMs perform less than satisfactorily on our
benchmark, demonstrating the scientific knowledge gap of existing LLMs.

这篇论文介绍了一种在核能领域评估语言模型性能的人工测试标准 ——NuclearQA，它由专家设计的 100 个问题组成，针对语言模型的能力进行了测试。论文还提出了一种新的评估指标，发现现有的最优语言模型在该测试标准上的表现不尽人意，揭示了现有语言模型的科学知识差距。