Big models, exemplified by Large Language Models (LLMs), are models typically
pre-trained on massive data and comprised of enormous parameters, which not
only obtain significantly improved performance across diverse tasks but also
present emergent capabilities absent in smaller models. However, the growing
intertwining of big models with everyday human lives poses potential risks and
might cause serious social harm. Therefore, many efforts have been made to
align LLMs with humans to make them better follow user instructions and satisfy
human preferences. Nevertheless, `what to align with' has not been fully
discussed, and inappropriate alignment goals might even backfire. In this
paper, we conduct a comprehensive survey of different alignment goals in
existing work and trace their evolution paths to help identify the most
essential goal. Particularly, we investigate related works from two
perspectives: the definition of alignment goals and alignment evaluation. Our
analysis encompasses three distinct levels of alignment goals and reveals a
goal transformation from fundamental abilities to value orientation, indicating
the potential of intrinsic human values as the alignment goal for enhanced
LLMs. Based on such results, we further discuss the challenges of achieving
such intrinsic value alignment and provide a collection of available resources
for future research on the alignment of big models.

通过综合调查现有工作的不同对齐目标并追踪其演变路径，本文揭示了从基本能力到价值取向的目标转变，表明内在人类价值可能是提升大型语言模型对齐目标的关键，进一步讨论了实现此内在价值对齐的挑战，并提供了一系列可用资源以支持未来对大型模型对齐的研究。