The recent development of generative and large language models (LLMs) poses
new challenges for model evaluation that the research community and industry
are grappling with. While the versatile capabilities of these models ignite
excitement, they also inevitably make a leap toward homogenization: powering a
wide range of applications with a single, often referred to as
``general-purpose'', model. In this position paper, we argue that model
evaluation practices must take on a critical task to cope with the challenges
and responsibilities brought by this homogenization: providing valid
assessments for whether and how much human needs in downstream use cases can be
satisfied by the given model (\textit{socio-technical gap}). By drawing on
lessons from the social sciences, human-computer interaction (HCI), and the
interdisciplinary field of explainable AI (XAI), we urge the community to
develop evaluation methods based on real-world socio-requirements and embrace
diverse evaluation methods with an acknowledgment of trade-offs between realism
to socio-requirements and pragmatic costs. By mapping HCI and current NLG
evaluation methods, we identify opportunities for new evaluation methods for
LLMs to narrow the socio-technical gap and pose open questions.

在这篇论文中，作者认为生成模型和大型语言模型（LLM）的发展给模型评估带来了新的挑战与责任，因此呼吁研究人员应开发基于真实世界的社会需求的评估方法，并兼顾现实与成本的平衡，以缩小模型使用中的社会技术差距，同时为大型语言模型的评估提供开放性问题和机会。