Generative Adversarial Networks (GANs) are a promising approach to language generation. The latest works introducing novel GAN models for language generation use n-gram based metrics for evaluation and only report single scores of the best run. In this paper, we argue that this often misrepresents the true picture and does not tell the full story, as GAN models can be extremely sensitive to the random initialization and small deviations from the best hyperparameter choice. In particular, we demonstrate that the previously used BLEU score is not sensitive to semantic deterioration of generated texts and propose alternative metrics that better capture the quality and diversity of the generated samples. We also conduct a set of experiments comparing a number of GAN models for text with a conventional Language Model (LM) and find that neither of the considered models performs convincingly better than the LM.

本文讨论了用于语言生成的生成对抗网络(GANs)模型的评估指标问题，认为现有的评估指标无法全面准确地反映模型更真实的表现，提出了一些更适合从质量和多样性方面评估GAN模型的新指标，并通过实验证明了相比传统语言模型(LM)，目前的GAN模型在语言生成方面的表现并没有明显优势。

准确评估GAN在语言生成中的性能