Class-based language models (LMs) have been long devised to address context
sparsity in $n$-gram LMs. In this study, we revisit this approach in the
context of neural LMs. We hypothesize that class-based prediction leads to an
implicit context aggregation for similar words and thus can improve
generalization for rare words. We map words that have a common WordNet hypernym
to the same class and train large neural LMs by gradually annealing from
predicting the class to token prediction during training. Empirically, this
curriculum learning strategy consistently improves perplexity over various
large, highly-performant state-of-the-art Transformer-based models on two
datasets, WikiText-103 and Arxiv. Our analysis shows that the performance
improvement is achieved without sacrificing performance on rare words. Finally,
we document other attempts that failed to yield empirical gains, and discuss
future directions for the adoption of class-based LMs on a larger scale.

通过将具有相同 WordNet 超类的单词映射到同一类中，并逐渐从预测类逐步训练为预测单词，我们在两个数据集上证明了该课程学习策略能够显著提高困惑度而不影响罕见词性能。

基于上位词类别预测的更好语言模型

Better Language Model with Hypernym Class Prediction

End-to-end modeling (E2E) of automatic speech recognition (ASR) blends all
the components of a traditional speech recognition system into a unified model.
Although it simplifies training and decoding pipelines, the unified model is
hard to adapt when mismatch exists between training and test data. In this
work, we focus on contextual speech recognition, which is particularly
challenging for E2E models because it introduces significant mismatch between
training and test data. To improve the performance in the presence of complex
contextual information, we propose to use class-based language models(CLM) that
can populate the classes with contextdependent information in real-time. To
enable this approach to scale to a large number of class members and minimize
search errors, we propose a token passing decoder with efficient token
recombination for E2E systems for the first time. We evaluate the proposed
system on general and contextual ASR, and achieve relative 62% Word Error
Rate(WER) reduction for contextual ASR without hurting performance for general
ASR. We show that the proposed method performs well without modification of the
decoding hyper-parameters across tasks, making it a general solution for E2E
ASR.

本文针对上下文语音识别难以适应 E2E 自动语音识别的问题，提出了使用基于类别的语言模型及高效微调令牌传递解码器来提高性能。实验表明，该方法在不改变解码超参数的情况下，有效降低了上下文语音识别的字词错误率，并不影响普通语音识别的表现。