Breakthroughs in deep learning and memory networks have made major advances
in natural language understanding. Language is sequential and information
carried through the sequence can be captured through memory networks. Learning
the sequence is one of the key aspects in learning the language. However,
memory networks are not capable of holding infinitely long sequences in their
memories and are limited by various constraints such as the vanishing or
exploding gradient problem. Therefore, natural language understanding models
are affected when presented with long sequential text. We introduce Long Term
Memory network (LTM) to learn from infinitely long sequences. LTM gives
priority to the current inputs to allow it to have a high impact. Language
modeling is an important factor in natural language understanding. LTM was
tested in language modeling, which requires long term memory. LTM is tested on
Penn Tree bank dataset, Google Billion Word dataset and WikiText-2 dataset. We
compare LTM with other language models which require long term memory.

本文介绍了长期记忆网络 (LTM) 以解决自然语言理解模型在处理长的序列文本时出现的问题。通过在语言建模任务上进行测试，我们发现 LTM 能够学习到无限长的序列信息，并与其他需要长时间记忆的语言模型进行比较。