Artificial intelligence (AI) has witnessed an evolution from task-specific to
general-purpose systems that trend toward human versatility. As AI systems
begin to play pivotal roles in society, it is important to ensure that they are
adequately evaluated. Current AI benchmarks typically assess performance on
collections of specific tasks. This has drawbacks when used for assessing
general-purpose AI systems. First, it is difficult to predict whether AI
systems could complete a new task it has never seen or that did not previously
exist. Second, these benchmarks often focus on overall performance metrics,
potentially overlooking the finer details crucial for making informed
decisions. Lastly, there are growing concerns about the reliability of existing
benchmarks and questions about what is being measured. To solve these
challenges, this paper suggests that psychometrics, the science of
psychological measurement, should be placed at the core of evaluating
general-purpose AI. Psychometrics provides a rigorous methodology for
identifying and measuring the latent constructs that underlie performance
across multiple tasks. We discuss its merits, warn against potential pitfalls,
and propose a framework for putting it into practice. Finally, we explore
future opportunities to integrate psychometrics with AI.

人工智能（AI）从任务特定到通用系统的发展，逐渐趋向于与人类的多样性。为了确保 AI 系统得到充分的评估，本文建议将心理测量学作为评估通用 AI 的核心，并提出相应的框架。

用心理测量评估通用人工智能

Evaluating General-Purpose AI with Psychometrics

Developing safe and useful general-purpose AI systems will require us to make
progress on scalable oversight: the problem of supervising systems that
potentially outperform us on most skills relevant to the task at hand.
Empirical work on this problem is not straightforward, since we do not yet have
systems that broadly exceed our abilities. This paper discusses one of the
major ways we think about this problem, with a focus on ways it can be studied
empirically. We first present an experimental design centered on tasks for
which human specialists succeed but unaided humans and current general AI
systems fail. We then present a proof-of-concept experiment meant to
demonstrate a key feature of this experimental design and show its viability
with two question-answering tasks: MMLU and time-limited QuALITY. On these
tasks, we find that human participants who interact with an unreliable
large-language-model dialog assistant through chat -- a trivial baseline
strategy for scalable oversight -- substantially outperform both the model
alone and their own unaided performance. These results are an encouraging sign
that scalable oversight will be tractable to study with present models and
bolster recent findings that large language models can productively assist
humans with difficult tasks.

本篇论文讨论了监督强于人类水平的 AI 系统的问题，提出了一个实验设计并探讨如何通过与打破传统 AI 的对话式助手交互的方式来解决这个问题。在基于两个问答任务进行的试验中，我们发现，通过这种方式监管的人类表现显著优于只使用大语言模型或人类自身的表现。