AI alignment aims to build AI systems that are in accordance with human
intentions and values. With the emergence of AI systems possessing superhuman
capabilities, the potential large-scale risks associated with misaligned
systems become apparent. Hundreds of AI experts and public figures have
expressed their concerns about AI risks, arguing that mitigating the risk of
extinction from AI should be a global priority, alongside other societal-scale
risks such as pandemics and nuclear war. Motivated by the lack of an up-to-date
systematic survey on AI alignment, in this paper, we delve into the core
concepts, methodology, and practice of alignment research. To begin with, we
identify four principles as the key objectives of AI alignment: Robustness,
Interpretability, Controllability, and Ethicality (RICE). We outline the
landscape of current alignment research and decompose them into two key
components: forward alignment and backward alignment. The former aims to make
AI systems aligned via alignment training, while the latter aims to gain
evidence about the systems' alignment and govern them appropriately to avoid
exacerbating misalignment risks. On forward alignment, we discuss how to
conduct learning from various types of feedback (a.k.a., outer alignment) and
how to overcome the distribution shift to avoid goal misgeneralization (a.k.a.,
inner alignment). On backward alignment, we discuss verification techniques
that can tell the degree of value alignment for various AI systems deployed,
which can further improve the assurance of forward alignment outcomes.
Based on this, we also release a constantly updated website featuring
tutorials, collections of papers, blogs, and other learning resources at
this https URL

AI alignment aims to build AI systems in accordance with human intentions and values, addressing the risks of misaligned systems with superhuman capabilities through forward and backward alignment methodologies.