Recent advancements in general-purpose AI have highlighted the importance of
guiding AI systems towards the intended goals, ethical principles, and values
of individuals and groups, a concept broadly recognized as alignment. However,
the lack of clarified definitions and scopes of human-AI alignment poses a
significant obstacle, hampering collaborative efforts across research domains
to achieve this alignment. In particular, ML- and philosophy-oriented alignment
research often views AI alignment as a static, unidirectional process (i.e.,
aiming to ensure that AI systems' objectives match humans) rather than an
ongoing, mutual alignment problem [429]. This perspective largely neglects the
long-term interaction and dynamic changes of alignment. To understand these
gaps, we introduce a systematic review of over 400 papers published between
2019 and January 2024, spanning multiple domains such as Human-Computer
Interaction (HCI), Natural Language Processing (NLP), Machine Learning (ML),
and others. We characterize, define and scope human-AI alignment. From this, we
present a conceptual framework of "Bidirectional Human-AI Alignment" to
organize the literature from a human-centered perspective. This framework
encompasses both 1) conventional studies of aligning AI to humans that ensures
AI produces the intended outcomes determined by humans, and 2) a proposed
concept of aligning humans to AI, which aims to help individuals and society
adjust to AI advancements both cognitively and behaviorally. Additionally, we
articulate the key findings derived from literature analysis, including
discussions about human values, interaction techniques, and evaluations. To
pave the way for future studies, we envision three key challenges for future
directions and propose examples of potential future solutions.

近期的人工智能的普适性进展凸显出引导人工智能系统走向个人和群体的目标、道德原则和价值观的重要性，这被广泛称为对齐。然而，人工智能和哲学领域对于人工智能和人类对齐的定义和范围尚未被明确界定，而且这个问题的长期互动和动态变化也基本被忽视。为了填补这些空白，我们对 2019 年至 2024 年 1 月间发表的 400 多篇论文进行了系统综述，涵盖了人机交互、自然语言处理、机器学习等多个领域。我们对人工智能与人类对齐进行了描述、定义和范围界定，并从以人为中心的视角提出了一个 “双向人工智能与人类对齐” 的概念框架来组织文献。该框架包括了将人工智能与人类对齐的传统研究，即确保人工智能系统产生由人类确定的预期结果，以及调整人类与人工智能对齐的概念，旨在帮助个人和社会在认知和行为上适应人工智能进展。此外，我们阐述了文献分析的关键发现，包括关于人类价值观、交互技术和评估的讨论。为了未来研究的发展，我们展望了未来方向的三个关键挑战，并提出了潜在的未来解决方案的例子。

朝双向人工智能对齐迈进：澄清、框架和未来发展的系统综述

Towards Bidirectional Human-AI Alignment: A Systematic Review for  Clarifications, Framework, and Future Directions

Achieving human-AI alignment in complex multi-agent games is crucial for
creating trustworthy AI agents that enhance gameplay. We propose a method to
evaluate this alignment using an interpretable task-sets framework, focusing on
high-level behavioral tasks instead of low-level policies. Our approach has
three components. First, we analyze extensive human gameplay data from Xbox's
Bleeding Edge (100K+ games), uncovering behavioral patterns in a complex task
space. This task space serves as a basis set for a behavior manifold capturing
interpretable axes: fight-flight, explore-exploit, and solo-multi-agent.
Second, we train an AI agent to play Bleeding Edge using a Generative
Pretrained Causal Transformer and measure its behavior. Third, we project human
and AI gameplay to the proposed behavior manifold to compare and contrast. This
allows us to interpret differences in policy as higher-level behavioral
concepts, e.g., we find that while human players exhibit variability in
fight-flight and explore-exploit behavior, AI players tend towards uniformity.
Furthermore, AI agents predominantly engage in solo play, while humans often
engage in cooperative and competitive multi-agent patterns. These stark
differences underscore the need for interpretable evaluation, design, and
integration of AI in human-aligned applications. Our study advances the
alignment discussion in AI and especially generative AI research, offering a
measurable framework for interpretable human-agent alignment in multiplayer
gaming.

评估人工智能与人类在复杂多智体游戏中的一致性，提出了一种使用可解释的任务集框架评估的方法，重点关注高级行为任务而非低级策略。通过分析人类游戏数据和训练 AI 代理，将人类和 AI 游戏对比，发现人类和 AI 之间存在显著差异，强调在多人游戏中应对人工智能的评估、设计和整合进行清晰解释。