In the study, we empirically compare the two recently proposed decoding
methods, i.e. Contrastive Search (CS) and Contrastive Decoding (CD), for
open-ended text generation. The automatic evaluation results suggest that,
while CS performs worse than CD on the MAUVE metric, it substantially surpasses
CD on the diversity and coherence metrics. More notably, extensive human
evaluations across three different domains demonstrate that human annotators
are universally more in favor of CS over CD with substantial margins.
The contradicted results between MAUVE and human evaluations reveal that
MAUVE does not accurately reflect human preferences. Therefore, we call upon
the research community to develop better evaluation metrics for open-ended text
generation. To ensure the reproducibility of our work, we have open-sourced all
our code, evaluation results, as well as human annotations at
this https URL.

本研究比较了两种最近提出的解码方法，即对比搜索（CS）和对比解码（CD），发现尽管 CS 在 MAUVE 指标上表现不佳，但在多样性和连贯性指标上大大超过 CD。人工评估显示，人类注释器普遍更喜欢 CS 而非 CD，MAUVE 与人工评估之间的矛盾表明 MAUVE 不准确反映人类偏好，需要开发更好的评估指标来确保生成结果的可复制性。

对比式搜索与对比式解码在开放式文本生成中的实证研究

An Empirical Study On Contrastive Search And Contrastive Decoding For Open-ended Text Generation

Generating text with autoregressive language models (LMs) is of great
importance to many natural language processing (NLP) applications. Previous
solutions for this task often produce text that contains degenerative
expressions or lacks semantic consistency. Recently, Su et al. introduced a new
decoding method, contrastive search, based on the isotropic representation
space of the language model and obtained new state of the art on various
benchmarks. Additionally, Su et al. argued that the representations of
autoregressive LMs (e.g. GPT-2) are intrinsically anisotropic which is also
shared by previous studies. Therefore, to ensure the language model follows an
isotropic distribution, Su et al. proposed a contrastive learning scheme,
SimCTG, which calibrates the language model's representations through
additional training.
In this study, we first answer the question: "Are autoregressive LMs really
anisotropic?". To this end, we extensively evaluate the isotropy of LMs across
16 major languages. Surprisingly, we find that the anisotropic problem only
exists in the two specific English GPT-2-small/medium models. On the other
hand, all other evaluated LMs are naturally isotropic which is in contrast to
the conclusion drawn by previous studies. Based on our findings, we further
assess the contrastive search decoding method using off-the-shelf LMs on four
generation tasks across 16 languages. Our experimental results demonstrate that
contrastive search significantly outperforms previous decoding methods without
any additional training. More notably, on 12 out of the 16 evaluated languages,
contrastive search performs comparably with human-level performances as judged
by human evaluations. Our code and other related resources are publicly
available at this https URL.

本研究评估了 16 种主要语言的自回归 LMs 的各向同性，并推出一个校准自回归 LMs 表示的对比学习方案，并基于此方案提出了一种新的解码方法，称为对比搜索，它在各种各样的基准测试中取得了最新的成果，显示出其高度的性能，无需任何附加的训练，其中在 16 个被评估的语言中有 12 个语种比人类评估得分相当。

神经文本生成需求对比搜索

Contrastive Search Is What You Need For Neural Text Generation

Text generation is of great importance to many natural language processing
applications. However, maximization-based decoding methods (e.g. beam search)
of neural language models often lead to degenerate solutions -- the generated
text is unnatural and contains undesirable repetitions. Existing approaches
introduce stochasticity via sampling or modify training objectives to decrease
probabilities of certain tokens (e.g., unlikelihood training). However, they
often lead to solutions that lack coherence. In this work, we show that an
underlying reason for model degeneration is the anisotropic distribution of
token representations. We present a contrastive solution: (i) SimCTG, a
contrastive training objective to calibrate the model's representation space,
and (ii) a decoding method -- contrastive search -- to encourage diversity
while maintaining coherence in the generated text. Extensive experiments and
analyses on three benchmarks from two languages demonstrate that our proposed
approach significantly outperforms current state-of-the-art text generation
methods as evaluated by both human and automatic metrics.

提出了一个新的基于对比学习的方法 ——SimCTG，和一个解码方法 —— 对比搜索，用于提高文本生成质量。在两种语言的三个基准测试上，我们的方法在人类和自动评估指标方面均显著优于当前最先进的文本生成方法。