Big models have achieved revolutionary breakthroughs in the field of AI, but
they might also pose potential concerns. Addressing such concerns, alignment
technologies were introduced to make these models conform to human preferences
and values. Despite considerable advancements in the past year, various
challenges lie in establishing the optimal alignment strategy, such as data
cost and scalable oversight, and how to align remains an open question. In this
survey paper, we comprehensively investigate value alignment approaches. We
first unpack the historical context of alignment tracing back to the 1920s
(where it comes from), then delve into the mathematical essence of alignment
(what it is), shedding light on the inherent challenges. Following this
foundation, we provide a detailed examination of existing alignment methods,
which fall into three categories: Reinforcement Learning, Supervised
Fine-Tuning, and In-context Learning, and demonstrate their intrinsic
connections, strengths, and limitations, helping readers better understand this
research area. In addition, two emerging topics, personal alignment, and
multimodal alignment, are also discussed as novel frontiers in this field.
Looking forward, we discuss potential alignment paradigms and how they could
handle remaining challenges, prospecting where future alignment will go.

大型模型在人工智能领域取得了革命性突破，但也可能引发一些潜在的担忧。本文综合研究价值对齐方法，探究了历史背景、数学本质以及现有对齐方法（强化学习、监督微调和上下文学习）的联系、优势和局限，并讨论了个性对齐和多模态对齐作为该领域的新兴方向。最后，展望了未来对齐范式以及如何处理剩余挑战。