Current AI alignment methodologies rely on human-provided demonstrations or
judgments, and the learned capabilities of AI systems would be upper-bounded by
human capabilities as a result. This raises a challenging research question:
How can we keep improving the systems when their capabilities have surpassed
the levels of humans? This paper answers this question in the context of
tackling hard reasoning tasks (e.g., level 4-5 MATH problems) via learning from
human annotations on easier tasks (e.g., level 1-3 MATH problems), which we
term as \textit{easy-to-hard generalization}. Our key insight is that an
evaluator (reward model) trained on supervisions for easier tasks can be
effectively used for scoring candidate solutions of harder tasks and hence
facilitating easy-to-hard generalization over different levels of tasks. Based
on this insight, we propose a novel approach to scalable alignment, which
firstly trains the process-supervised reward models on easy problems (e.g.,
level 1-3), and then uses them to evaluate the performance of policy models on
hard problems. We show that such \textit{easy-to-hard generalization from
evaluators} can enable \textit{easy-to-hard generalizations in generators}
either through re-ranking or reinforcement learning (RL). Notably, our
process-supervised 7b RL model achieves an accuracy of 34.0\% on MATH500,
despite only using human supervision on easy problems. Our approach suggests a
promising path toward AI systems that advance beyond the frontier of human
supervision.

通过从易到难的泛化和评估者的使用，本文提出一种可扩展的 AI 对齐方法，用于解决超越人类监督水平的困难推理任务，提升生成器模型在数学问题上的准确率。

易于困难泛化：超越人类监督的可扩展对齐

Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision

How can we train models to perform well on hard test data when hard training
data is by definition difficult to label correctly? This question has been
termed the scalable oversight problem and has drawn increasing attention as
language models have continually improved. In this paper, we present the
surprising conclusion that current language models often generalize relatively
well from easy to hard data, even performing as well as "oracle" models trained
on hard data. We demonstrate this kind of easy-to-hard generalization using
simple training methods like in-context learning, linear classifier heads, and
QLoRA for seven different measures of datapoint hardness, including six
empirically diverse human hardness measures (like grade level) and one
model-based measure (loss-based). Furthermore, we show that even if one cares
most about model performance on hard data, it can be better to collect and
train on easy data rather than hard data, since hard data is generally noisier
and costlier to collect. Our experiments use open models up to 70b in size and
four publicly available question-answering datasets with questions ranging in
difficulty from 3rd grade science questions to college level STEM questions and
general-knowledge trivia. We conclude that easy-to-hard generalization in LMs
is surprisingly strong for the tasks studied, suggesting the scalable oversight
problem may be easier than previously thought. Our code is available at
this https URL

通过对容易和困难数据进行简单的训练方法、线性分类器头和 QLoRA 的易变难泛化，以及使用不同硬度度量的实验验证，我们得出了在语言模型中易变难泛化意外地很强，表明可扩展的监管问题可能比之前认为的更容易。