Recurrent neural network (RNN) based character-level language models (CLMs)
are extremely useful for modeling out-of-vocabulary words by nature. However,
their performance is generally much worse than the word-level language models
(WLMs), since CLMs need to consider longer history of tokens to properly
predict the next one. We address this problem by proposing hierarchical RNN
architectures, which consist of multiple modules with different timescales.
Despite the multi-timescale structures, the input and output layers operate
with the character-level clock, which allows the existing RNN CLM training
approaches to be directly applicable without any modifications. Our CLM models
show better perplexity than Kneser-Ney (KN) 5-gram WLMs on the One Billion Word
Benchmark with only 2% of parameters. Also, we present real-time
character-level end-to-end speech recognition examples on the Wall Street
Journal (WSJ) corpus, where replacing traditional mono-clock RNN CLMs with the
proposed models results in better recognition accuracies even though the number
of parameters are reduced to 30%.

我们通过提出分层 RNN 体系结构，提高了基于字符级语言模型的溢出词建模的性能。在 One Billion Word Benchmark 上，我们的 CLM 模型比 Kneser-Ney 5 元语言模型表现更好，同时参数却只有 2%。我们在 WSJ 语料库上展示了对话识别示例，并将传统的 RNN CLMs 替换为所提出的模型，即使参数数量减少了 30％，精度也有所提高。

层次循环神经网络的字符级语言模型

Character-Level Language Modeling with Hierarchical Recurrent Neural  Networks

In this work we explore recent advances in Recurrent Neural Networks for
large scale Language Modeling, a task central to language understanding. We
extend current models to deal with two key challenges present in this task:
corpora and vocabulary sizes, and complex, long term structure of language. We
perform an exhaustive study on techniques such as character Convolutional
Neural Networks or Long-Short Term Memory, on the One Billion Word Benchmark.
Our best single model significantly improves state-of-the-art perplexity from
51.3 down to 30.0 (whilst reducing the number of parameters by a factor of 20),
while an ensemble of models sets a new record by improving perplexity from 41.0
down to 23.7. We also release these models for the NLP and ML community to
study and improve upon.

本文研究了递归神经网络在大规模语言建模中的最新进展和应用，对语料库和词汇量的大小和语言的复杂性和长期结构等问题进行了探讨，并在 One Billion Word Benchmark 上进行了详尽的研究，最佳单一模型将习惯度从 51.3 降低到 30.0，而模型集成则创下了 41.0 到 23.7 的新纪录，在总结中，研究结果可供自然语言处理和机器学习界进一步研究和提高。