Preference optimization is a standard approach to fine-tuning large language
models to align with human preferences. The quality, diversity, and quantity of
the preference dataset are critical to the effectiveness of preference
optimization. However, obtaining a large amount of high-quality and diverse
preference annotations is difficult in many applications. This raises the
question of how to use the limited annotation budget to create an effective
preference dataset. To this end, we propose Annotation-Efficient Preference
Optimization (AEPO). Instead of exhaustively annotating preference over all
available response texts, AEPO selects a subset of responses that maximizes
quality and diversity from the available responses, and then annotates
preference over the selected ones. In this way, AEPO focuses the annotation
budget on labeling preference over a smaller subset of responses with diversity
and of high quality. We evaluate the performance of Direct Preference
Optimization (DPO) using AEPO and show that it outperforms models trained using
a standard DPO with the same annotation budget. Our code is available at
this https URL

提出了 Annotation-Efficient Preference Optimization (AEPO) 的方法，通过选择能最大化质量和多样性的回应子集并对之进行偏好标注，将有限的标注预算集中用于标注质量高且多样性较好的较小回应子集上，比标准的 Direct Preference Optimization (DPO) 方法在同样的标注预算下表现更好。

语言模型对齐的注释高效优化

Annotation-Efficient Preference Optimization for Language Model  Alignment

News headline generation aims to produce a short sentence to attract readers
to read the news. One news article often contains multiple keyphrases that are
of interest to different users, which can naturally have multiple reasonable
headlines. However, most existing methods focus on the single headline
generation. In this paper, we propose generating multiple headlines with
keyphrases of user interests, whose main idea is to generate multiple
keyphrases of interest to users for the news first, and then generate multiple
keyphrase-relevant headlines. We propose a multi-source Transformer decoder,
which takes three sources as inputs: (a) keyphrase, (b) keyphrase-filtered
article, and (c) original article to generate keyphrase-relevant, high-quality,
and diverse headlines. Furthermore, we propose a simple and effective method to
mine the keyphrases of interest in the news article and build a first
large-scale keyphrase-aware news headline corpus, which contains over 180K
aligned triples of $<$news article, headline, keyphrase$>$. Extensive
experimental comparisons on the real-world dataset show that the proposed
method achieves state-of-the-art results in terms of quality and diversity

本文提出了一种生成多个与关键词相关、高质量、多样化新闻标题的方法，使用多源 Transformer 解码器，并构建了第一个大规模的具有关键词意识的新闻标题语料库，并在真实数据集上进行了广泛实验比较。

多样化、可控、关键词感知：新闻多标题生成的语料库和方法

Diverse, Controllable, and Keyphrase-Aware: A Corpus and Method for News  Multi-Headline Generation

Generative Adversarial Networks (GANs) are a promising approach to language
generation. The latest works introducing novel GAN models for language
generation use n-gram based metrics for evaluation and only report single
scores of the best run. In this paper, we argue that this often misrepresents
the true picture and does not tell the full story, as GAN models can be
extremely sensitive to the random initialization and small deviations from the
best hyperparameter choice. In particular, we demonstrate that the previously
used BLEU score is not sensitive to semantic deterioration of generated texts
and propose alternative metrics that better capture the quality and diversity
of the generated samples. We also conduct a set of experiments comparing a
number of GAN models for text with a conventional Language Model (LM) and find
that neither of the considered models performs convincingly better than the LM.

本文讨论了用于语言生成的生成对抗网络 (GANs) 模型的评估指标问题，认为现有的评估指标无法全面准确地反映模型更真实的表现，提出了一些更适合从质量和多样性方面评估 GAN 模型的新指标，并通过实验证明了相比传统语言模型 (LM)，目前的 GAN 模型在语言生成方面的表现并没有明显优势。