Personalized text generation presents a specialized mechanism for delivering
content that is specific to a user's personal context. While the research
progress in this area has been rapid, evaluation still presents a challenge.
Traditional automated metrics such as BLEU and ROUGE primarily measure lexical
similarity to human-written references, and are not able to distinguish
personalization from other subtle semantic aspects, thus falling short of
capturing the nuances of personalized generated content quality. On the other
hand, human judgments are costly to obtain, especially in the realm of
personalized evaluation. Inspired by these challenges, we explore the use of
large language models (LLMs) for evaluating personalized text generation, and
examine their ability to understand nuanced user context. We present AuPEL, a
novel evaluation method that distills three major semantic aspects of the
generated text: personalization, quality and relevance, and automatically
measures these aspects. To validate the effectiveness of AuPEL, we design
carefully controlled experiments and compare the accuracy of the evaluation
judgments made by LLMs versus that of judgements made by human annotators, and
conduct rigorous analyses of the consistency and sensitivity of the proposed
metric. We find that, compared to existing evaluation metrics, AuPEL not only
distinguishes and ranks models based on their personalization abilities more
accurately, but also presents commendable consistency and efficiency for this
task. Our work suggests that using LLMs as the evaluators of personalized text
generation is superior to traditional text similarity metrics, even though
interesting new challenges still remain.

个性化文本生成采用了一种专门的机制来提供与用户个人背景相关的内容，并且尽管这一领域的研究进展很快，但仍存在着评估方面的挑战。本研究提出了 AuPEL 作为一种新颖的评估方法，通过大型语言模型对个性化文本生成进行评估，用以满足人工评估的高成本和传统文本相似度度量方法的不足。实验证明，与现有的评估指标相比，AuPEL 不仅可以更准确地区分和排名模型的个性化能力，而且在此任务中表现出令人称赞的一致性和高效性。

基于大型语言模型的个性化文本生成的自动评估

Automated Evaluation of Personalized Text Generation using Large  Language Models

Facilitated by large language models (LLMs), personalized text generation has
become a rapidly growing research direction. Most existing studies focus on
designing specialized models for a particular domain, or they require
fine-tuning the LLMs to generate personalized text. We consider a typical
scenario in which the large language model, which generates personalized
output, is frozen and can only be accessed through APIs. Under this constraint,
all one can do is to improve the input text (i.e., text prompts) sent to the
LLM, a procedure that is usually done manually. In this paper, we propose a
novel method to automatically revise prompts for personalized text generation.
The proposed method takes the initial prompts generated by a state-of-the-art,
multistage framework for personalized generation and rewrites a few critical
components that summarize and synthesize the personal context. The prompt
rewriter employs a training paradigm that chains together supervised learning
(SL) and reinforcement learning (RL), where SL reduces the search space of RL
and RL facilitates end-to-end training of the rewriter. Using datasets from
three representative domains, we demonstrate that the rewritten prompts
outperform both the original prompts and the prompts optimized via supervised
learning or reinforcement learning alone. In-depth analysis of the rewritten
prompts shows that they are not only human readable, but also able to guide
manual revision of prompts when there is limited resource to employ
reinforcement learning to train the prompt rewriter, or when it is costly to
deploy an automatic prompt rewriter for inference.

通过自动修订提示符来进行个性化文本生成的研究。

个性化文本生成的自动提示改写

Automatic Prompt Rewriting for Personalized Text Generation

Personalized text generation is an emerging research area that has attracted
much attention in recent years. Most studies in this direction focus on a
particular domain by designing bespoke features or models. In this work, we
propose a general approach for personalized text generation using large
language models (LLMs). Inspired by the practice of writing education, we
develop a multistage and multitask framework to teach LLMs for personalized
generation. In writing instruction, the task of writing from sources is often
decomposed into multiple steps that involve finding, evaluating, summarizing,
synthesizing, and integrating information. Analogously, our approach to
personalized text generation consists of multiple stages: retrieval, ranking,
summarization, synthesis, and generation. In addition, we introduce a multitask
setting that helps the model improve its generation ability further, which is
inspired by the observation in education that a student's reading proficiency
and writing ability are often correlated. We evaluate our approach on three
public datasets, each of which covers a different and representative domain.
Our results show significant improvements over a variety of baselines.

个性化文本生成是一个新兴的研究领域，通过使用大型语言模型 (LLMs)，我们提出了一种通用的个性化文本生成方法。借鉴写作教育的实践，我们开发了一个多阶段和多任务的框架来教授 LLMs 进行个性化生成。我们的方法包括检索、排名、总结、综合和生成多个阶段，并引入了一个多任务设置来进一步提高模型的生成能力。我们在三个涵盖不同代表性领域的公开数据集上评估了我们的方法，结果显示相对于各种基线模型，我们取得了显著的改进。

教导 LLM 个性化 —— 受到写作教育启发的方法

Teach LLMs to Personalize -- An Approach inspired by Writing Education

Personalized text generation has broad industrial applications, such as
explanation generation for recommendations, conversational systems, etc.
Personalized text generators are usually trained on user written text, e.g.,
reviews collected on e-commerce platforms. However, due to historical, social,
or behavioral reasons, there may exist bias that associates certain linguistic
quality of user written text with the users' protected attributes such as
gender, race, etc. The generators can identify and inherit these correlations
and generate texts discriminately w.r.t. the users' protected attributes.
Without proper intervention, such bias can adversarially influence the users'
trust and reliance on the system. From a broader perspective, bias in
auto-generated contents can reinforce the social stereotypes about how online
users write through interactions with the users.
In this work, we investigate the fairness of personalized text generation in
the setting of explainable recommendation. We develop a general framework for
achieving measure-specific counterfactual fairness on the linguistic quality of
personalized explanations. We propose learning disentangled representations for
counterfactual inference and develop a novel policy learning algorithm with
carefully designed rewards for fairness optimization. The framework can be
applied for achieving fairness on any given specifications of linguistic
quality measures, and can be adapted to most of existing models and real-world
settings. Extensive experiments demonstrate the superior ability of our method
in achieving fairness while maintaining high generation performance.

本文研究个性化文本生成中的公平性问题，提出了一个可适用于现有模型和实际场景的通用框架，以促进文本的公平性，消除因用户属性造成的偏见。