Aligning AI agents with human values is challenging due to diverse and subjective notions of values. Standard alignment methods often aggregate crowd feedback, which can result in the suppression of unique or minority preferences. We introduce Interactive-Reflective Dialogue Alignment, a method that iteratively engages users in reflecting on and specifying their subjective value definitions. This system learns individual value definitions through language-model-based preference elicitation and constructs personalized reward models that can be used to align AI behaviour. We evaluated our system through two studies with 30 participants, one focusing on "respect" and the other on ethical decision-making in autonomous vehicles. Our findings demonstrate diverse definitions of value-aligned behaviour and show that our system can accurately capture each person's unique understanding. This approach enables personalized alignment and can inform more representative and interpretable collective alignment strategies.

本文解决了人工智能代理与人类多样化价值观对齐的挑战，提出了一种互动反思对话对齐的方法，可以在用户反思和定义其主观价值的过程中迭代学习个体价值定义。研究表明，该方法能够准确捕捉个人对价值对齐行为的独特理解，从而实现个性化对齐，并启发更具代表性和可解释的集体对齐策略。

民主化个性化和代表性价值对齐的奖励设计