Recurrent neural network (RNN) based character-level language models (CLMs)
are extremely useful for modeling out-of-vocabulary words by nature. However,
their performance is generally much worse than the word-level language models
(WLMs), since CLMs need to consider longer history of tokens to properly
predict the next one. We address this problem by proposing hierarchical RNN
architectures, which consist of multiple modules with different timescales.
Despite the multi-timescale structures, the input and output layers operate
with the character-level clock, which allows the existing RNN CLM training
approaches to be directly applicable without any modifications. Our CLM models
show better perplexity than Kneser-Ney (KN) 5-gram WLMs on the One Billion Word
Benchmark with only 2% of parameters. Also, we present real-time
character-level end-to-end speech recognition examples on the Wall Street
Journal (WSJ) corpus, where replacing traditional mono-clock RNN CLMs with the
proposed models results in better recognition accuracies even though the number
of parameters are reduced to 30%.

我们通过提出分层 RNN 体系结构，提高了基于字符级语言模型的溢出词建模的性能。在 One Billion Word Benchmark 上，我们的 CLM 模型比 Kneser-Ney 5 元语言模型表现更好，同时参数却只有 2%。我们在 WSJ 语料库上展示了对话识别示例，并将传统的 RNN CLMs 替换为所提出的模型，即使参数数量减少了 30％，精度也有所提高。