We study interactive learning of language agents based on user edits made to
the agent's output. In a typical setting such as writing assistants, the user
interacts with a language agent to generate a response given a context, and may
optionally edit the agent response to personalize it based on their latent
preference, in addition to improving the correctness. The edit feedback is
naturally generated, making it a suitable candidate for improving the agent's
alignment with the user's preference, and for reducing the cost of user edits
over time. We propose a learning framework, PRELUDE that infers a description
of the user's latent preference based on historic edit data and using it to
define a prompt policy that drives future response generation. This avoids
fine-tuning the agent, which is costly, challenging to scale with the number of
users, and may even degrade its performance on other tasks. Furthermore,
learning descriptive preference improves interpretability, allowing the user to
view and modify the learned preference. However, user preference can be complex
and vary based on context, making it challenging to learn. To address this, we
propose a simple yet effective algorithm named CIPHER that leverages a large
language model (LLM) to infer the user preference for a given context based on
user edits. In the future, CIPHER retrieves inferred preferences from the
k-closest contexts in the history, and forms an aggregate preference for
response generation. We introduce two interactive environments -- summarization
and email writing, for evaluation using a GPT-4 simulated user. We compare with
algorithms that directly retrieve user edits but do not learn descriptive
preference, and algorithms that learn context-agnostic preference. On both
tasks, CIPHER achieves the lowest edit distance cost and learns preferences
that show significant similarity to the ground truth preferences

基于用户编辑的互动学习语言代理，通过历史编辑数据推断用户的潜在偏好，定义推动未来回复生成的提示策略，实现代理和用户偏好的对齐，减少用户编辑成本和提高性能。