Reinforcement learning from human feedback (RLHF) is fundamentally limited by
the capacity of humans to correctly evaluate model output. To improve human
evaluation ability and overcome that limitation this work trains "critic"
models that help humans to more accurately evaluate model-written code. These
critics are themselves LLMs trained with RLHF to write natural language
feedback highlighting problems in code from real-world assistant tasks. On code
containing naturally occurring LLM errors model-written critiques are preferred
over human critiques in 63% of cases, and human evaluation finds that models
catch more bugs than human contractors paid for code review. We further confirm
that our fine-tuned LLM critics can successfully identify hundreds of errors in
ChatGPT training data rated as "flawless", even though the majority of those
tasks are non-code tasks and thus out-of-distribution for the critic model.
Critics can have limitations of their own, including hallucinated bugs that
could mislead humans into making mistakes they might have otherwise avoided,
but human-machine teams of critics and contractors catch similar numbers of
bugs to LLM critics while hallucinating less than LLMs alone.

人类反馈强化学习受到人类正确评估模型输出能力的限制。为了提高人类评估能力并克服这一限制，本研究训练了 “评论家” 模型，帮助人类更准确地评估模型生成的代码。这些评论家是通过强化学习从人类反馈训练的语言模型，用于指出真实世界助手任务中代码中的问题。在含有自然发生的语言模型错误的代码中，63% 的情况下优先选择模型生成的评论，而人工评估发现模型比人类承包商在代码审核中能找到更多的错误。我们进一步验证了我们微调的语言模型评论家能够成功地在被评为 “无瑕疵” 的 ChatGPT 训练数据中识别出数百个错误，尽管绝大多数任务是非代码任务，因此对评论家模型来说是非分布的。评论家也可能存在其自身的局限性，包括产生错误的问题，可能会误导人类错误地做出本来可以避免的错误，但是人机评论家与承包商团队能够发现与仅使用语言模型评论相似数量的错误，并且比仅使用语言模型时产生更少的错误。

LLM 批评家助力捕捉 LLM 漏洞

LLM Critics Help Catch LLM Bugs

We explore the use of aggregative crowdsourced forecasting (ACF) as a
mechanism to help operationalize ``collective intelligence'' of human-machine
teams for coordinated actions. We adopt the definition for Collective
Intelligence as: ``A property of groups that emerges from synergies among
data-information-knowledge, software-hardware, and individuals (those with new
insights as well as recognized authorities) that enables just-in-time knowledge
for better decisions than these three elements acting alone.'' Collective
Intelligence emerges from new ways of connecting humans and AI to enable
decision-advantage, in part by creating and leveraging additional sources of
information that might otherwise not be included. Aggregative crowdsourced
forecasting (ACF) is a recent key advancement towards Collective Intelligence
wherein predictions (X\% probability that Y will happen) and rationales (why I
believe it is this probability that X will happen) are elicited independently
from a diverse crowd, aggregated, and then used to inform higher-level
decision-making. This research asks whether ACF, as a key way to enable
Operational Collective Intelligence, could be brought to bear on operational
scenarios (i.e., sequences of events with defined agents, components, and
interactions) and decision-making, and considers whether such a capability
could provide novel operational capabilities to enable new forms of
decision-advantage.

使用聚合性众包预测（ACF）作为一种机制来实现人机团队的 “集体智能” 的具体操作，以提供协调行动的能力，并探索 ACF 作为一种关键的方法，以增强决策能力和开启决策优势的新形式。

人机协同智能运营

Operational Collective Intelligence of Humans and Machines

A rising vision for AI in the open world centers on the development of
systems that can complement humans for perceptual, diagnostic, and reasoning
tasks. To date, systems aimed at complementing the skills of people have
employed models trained to be as accurate as possible in isolation. We
demonstrate how an end-to-end learning strategy can be harnessed to optimize
the combined performance of human-machine teams by considering the distinct
abilities of people and machines. The goal is to focus machine learning on
problem instances that are difficult for humans, while recognizing instances
that are difficult for the machine and seeking human input on them. We
demonstrate in two real-world domains (scientific discovery and medical
diagnosis) that human-machine teams built via these methods outperform the
individual performance of machines and people. We then analyze conditions under
which this complementarity is strongest, and which training methods amplify it.
Taken together, our work provides the first systematic investigation of how
machine learning systems can be trained to complement human reasoning.

本文针对开放环境中的 AI，通过使用端到端学习策略，结合人类和机器的不同能力组成的人机团队优化其组合性能，并在科学发现和医疗诊断领域进行了实证研究，结果表明表现优于单独的机器和人类认知，是对机器学习如何辅助人类推理的系统性研究。