Superalignment, where humans are weak supervisors of superhuman models, has
become an important and widely discussed issue in the current era of rapid
development of Large Language Models (LLMs). The recent work preliminarily
studies this problem by using weak models to supervise strong models. It
discovers that weakly supervised strong students can consistently outperform
weak teachers towards the alignment target, leading to a weak-to-strong
generalization phenomenon. However, we are concerned that behind such a
promising phenomenon, whether there exists an issue of weak-to-strong
deception, where strong models may deceive weak models by exhibiting
well-aligned in areas known to weak models but producing misaligned behaviors
in cases weak models do not know. We then take an initial step towards
exploring this security issue in a specific but realistic multi-objective
alignment case, where there may be some alignment targets conflicting with each
other (e.g., helpfulness v.s. harmlessness). Such a conflict is likely to cause
strong models to deceive weak models in one alignment dimension to gain high
reward in other alignment dimension. Our experiments on both the reward
modeling task and the preference optimization scenario indicate: (1) the
weak-to-strong deception exists; (2) the deception phenomenon may intensify as
the capability gap between weak and strong models increases. We also discuss
potential solutions and find bootstrapping with an intermediate model can
mitigate the deception to some extent. Our work highlights the urgent need to
pay more attention to the true reliability of superalignment.

通过使用具有弱监督的模型对强大模型进行监督，最近的研究初步探讨了超级对齐问题。实验发现，弱监督的强学生在对齐目标上持续胜过弱教师，引发了弱到强泛化现象。然而，我们担心在这种令人期待的现象背后，是否存在弱到强欺骗问题，即强大模型可能通过在弱模型已知领域中表现得很好，而在弱模型不知道的情况下产生不对齐的行为进行欺骗。我们在特定但现实的多目标对齐情况下以及奖励建模任务和偏好优化场景上的实验证明：（1）存在弱到强的欺骗现象；（2）随着弱模型和强模型能力差距的增加，欺骗现象可能会加剧。我们还讨论了潜在的解决方案，并发现通过中间模型的引导可以在一定程度上减轻欺骗问题。我们的工作强调了对超级对齐的真实可靠性更加紧迫的关注。