Large language models (LLMs) are now rapidly advancing and surpassing human
abilities on many natural language tasks. However, aligning these super-human
LLMs with human knowledge remains challenging because the supervision signals
from human annotators may be wrong. This issue, known as the "super-alignment"
problem, requires enhancing weak-to-strong generalization, where a strong LLM
must generalize from imperfect supervision provided by a weaker source. To
address this issue, we propose an approach to improve weak-to-strong
generalization by involving the reliability of weak supervision signals in the
alignment process. In our method, we query the weak supervisor for multiple
answers, estimate the answer reliability, and enhance the alignment process by
filtering out uncertain data or re-weighting reliable data. Experiments on four
datasets demonstrate that our methods effectively identify the quality of weak
labels and significantly enhance weak-to-strong generalization. Our work
presents effective techniques for error-robust model alignment, reducing error
propagation from noisy supervision and enhancing the accuracy and reliability
of LLMs. Codes are publicly available at
this http URL

通过提高弱监督信号的可靠性，我们的方法有效地识别了弱标签的质量，并显著提高了弱到强泛化能力，降低了噪声监督的误差传播，增强了大型语言模型的准确性和可靠性。