Language models are generally trained on data spanning a wide range of topics
(e.g., news, reviews, fiction), but they might be applied to an a priori
unknown target distribution (e.g., restaurant reviews). In this paper, we first
show that training on text outside the test distribution can degrade test
performance when using standard maximum likelihood (MLE) training. To remedy
this without the knowledge of the test distribution, we propose an approach
which trains a model that performs well over a wide range of potential test
distributions. In particular, we derive a new distributionally robust
optimization (DRO) procedure which minimizes the loss of the model over the
worst-case mixture of topics with sufficient overlap with the training
distribution. Our approach, called topic conditional value at risk (topic
CVaR), obtains a 5.5 point perplexity reduction over MLE when the language
models are trained on a mixture of Yelp reviews and news and tested only on
reviews.

本文研究语言模型在未知测试分布下的性能问题，并提出了一种基于分布鲁棒性的优化方法，称为主题条件风险值（topic CVaR），该方法能够在广泛的潜在测试分布下表现良好，并且在使用 Yelp 评论和新闻混合训练语言模型，并仅在评论数据上进行测试时，相对于标准最大似然估计法（MLE），实现了 5.5 点困惑度的降低。