This is a tutorial paper on Recurrent Neural Network (RNN), Long Short-Term
Memory Network (LSTM), and their variants. We start with a dynamical system and
backpropagation through time for RNN. Then, we discuss the problems of gradient
vanishing and explosion in long-term dependencies. We explain close-to-identity
weight matrix, long delays, leaky units, and echo state networks for solving
this problem. Then, we introduce LSTM gates and cells, history and variants of
LSTM, and Gated Recurrent Units (GRU). Finally, we introduce bidirectional RNN,
bidirectional LSTM, and the Embeddings from Language Model (ELMo) network, for
processing a sequence in both directions.

本研究论文介绍了递归神经网络 (RNN)、 长短期记忆网络 (LSTM) 及其变体，在训练过程中解决长序列中的梯度消失 / 爆炸问题，并详细介绍了 LSTM 中的门控单元，双向 RNN 及 Embeddings from Language Model (ELMo) network 的原理。

循环神经网络和长短时记忆网络：教程和调研

Recurrent Neural Networks and Long Short-Term Memory Networks: Tutorial  and Survey

We consider the problem of training input-output recurrent neural networks
(RNN) for sequence labeling tasks. We propose a novel spectral approach for
learning the network parameters. It is based on decomposition of the
cross-moment tensor between the output and a non-linear transformation of the
input, based on score functions. We guarantee consistent learning with
polynomial sample and computational complexity under transparent conditions
such as non-degeneracy of model parameters, polynomial activations for the
neurons, and a Markovian evolution of the input sequence. We also extend our
results to Bidirectional RNN which uses both previous and future information to
output the label at each time point, and is employed in many NLP tasks such as
POS tagging.

研究新颖的谱方法，通过得分函数将输出和输入的非线性变换之间的交叉矩张量分解，以解决逐步标记任务的输入 - 输出循环神经网络 (RNN) 的训练问题，并将其扩展到双向 RNN，该方法被应用于许多自然语言处理 (NLP) 任务，如 POS 标记。