Text generative models trained via Maximum Likelihood Estimation (MLE) suffer from the notorious exposure bias problem, and Generative Adversarial Networks (GANs) are shown to have potential to tackle it. Existing language GANs adopt estimators like REINFORCE or continuous relaxations to model word distributions. The inherent limitations of such estimators lead current models to rely on pre-training techniques (MLE pre-training or pre-trained embeddings). Representation modeling methods which are free from those limitations, however, are seldom explored because of its poor performance in previous attempts. Our analyses reveal that invalid sampling method and unhealthy gradients are the main contributors to its unsatisfactory performance. In this work, we present two techniques to tackle these problems: dropout sampling and fully normalized LSTM. Based on these two techniques, we propose InitialGAN whose parameters are randomly initialized completely. Besides, we introduce a new evaluation metric, Least Coverage Rate, to better evaluate the quality of generated samples. The experimental results demonstrate that InitialGAN outperforms both MLE and other compared models. To the best of our knowledge, it is the first time a language GAN can outperform MLE without any pre-training techniques.

本文介绍了两种技术(dropout采样和完全规范化的LSTM)，提出了InitialGAN模型，并引入了一个新的评估指标——最小覆盖率，试图解决最常遇到的暴露偏差问题，实验结果表明InitialGAN优于MLE和其他比较的模型，比我们所知任何一种语言GAN都可以不使用任何预训练技术超越MLE。

InitialGAN：一种完全随机初始化的语言生成对抗网络