Large Language Models (LLMs) are essential tools to collaborate with users on
different tasks. Evaluating their performance to serve users' needs in
real-world scenarios is important. While many benchmarks have been created,
they mainly focus on specific predefined model abilities. Few have covered the
intended utilization of LLMs by real users. To address this oversight, we
propose benchmarking LLMs from a user perspective in both dataset construction
and evaluation designs. We first collect 1863 real-world use cases with 15 LLMs
from a user study with 712 participants from 23 countries. These self-reported
cases form the User Reported Scenarios(URS) dataset with a categorization of 7
user intents. Secondly, on this authentic multi-cultural dataset, we benchmark
10 LLM services on their efficacy in satisfying user needs. Thirdly, we show
that our benchmark scores align well with user-reported experience in LLM
interactions across diverse intents, both of which emphasize the overlook of
subjective scenarios. In conclusion, our study proposes to benchmark LLMs from
a user-centric perspective, aiming to facilitate evaluations that better
reflect real user needs. The benchmark dataset and code are available at
this https URL

我们提出了从用户角度对大型语言模型进行基准测试，旨在更好地反映实际用户需求，并且我们还构建了用户报告场景数据集以及对 10 个语言模型服务在满足用户需求方面的基准测试。

一个以用户为中心的评估大型语言模型的基准

A User-Centric Benchmark for Evaluating Large Language Models

As a highly data-driven application, recommender systems could be affected by
data bias, resulting in unfair results for different data groups, which could
be a reason that affects the system performance. Therefore, it is important to
identify and solve the unfairness issues in recommendation scenarios. In this
paper, we address the unfairness problem in recommender systems from the user
perspective. We group users into advantaged and disadvantaged groups according
to their level of activity, and conduct experiments to show that current
recommender systems will behave unfairly between two groups of users.
Specifically, the advantaged users (active) who only account for a small
proportion in data enjoy much higher recommendation quality than those
disadvantaged users (inactive). Such bias can also affect the overall
performance since the disadvantaged users are the majority. To solve this
problem, we provide a re-ranking approach to mitigate this unfairness problem
by adding constraints over evaluation metrics. The experiments we conducted on
several real-world datasets with various recommendation algorithms show that
our approach can not only improve group fairness of users in recommender
systems, but also achieve better overall recommendation performance.

本文基于用户视角，探讨推荐系统中存在的不公平问题，提出使用重新排序的方法来减轻推荐系统在活跃用户和非活跃用户之间的质量偏差问题，从而提高系统的公平性和整体推荐性能。

面向用户的推荐公平性

User-oriented Fairness in Recommendation

In his seminal book `The Inmates are Running the Asylum: Why High-Tech
Products Drive Us Crazy And How To Restore The Sanity' [2004, Sams
Indianapolis, IN, USA], Alan Cooper argues that a major reason why software is
often poorly designed (from a user perspective) is that programmers are in
charge of design decisions, rather than interaction designers. As a result,
programmers design software for themselves, rather than for their target
audience, a phenomenon he refers to as the `inmates running the asylum'. This
paper argues that explainable AI risks a similar fate. While the re-emergence
of explainable AI is positive, this paper argues most of us as AI researchers
are building explanatory agents for ourselves, rather than for the intended
users. But explainable AI is more likely to succeed if researchers and
practitioners understand, adopt, implement, and improve models from the vast
and valuable bodies of research in philosophy, psychology, and cognitive
science, and if evaluation of these models is focused more on people than on
technology. From a light scan of literature, we demonstrate that there is
considerable scope to infuse more results from the social and behavioural
sciences into explainable AI, and present some key results from these fields
that are relevant to explainable AI.

该论文认为解释性人工智能（AI）存在一个程序员主导设计决策的类似问题，需要将哲学、心理学和认知科学等社会和行为科学的大量研究结果纳入可解释的 AI，从人的角度评估这些模型的质量，以取得更大的成功。