Recurrent neural networks (RNNs) provide state-of-the-art performance in
processing sequential data but are memory intensive to train, limiting the
flexibility of RNN models which can be trained. Reversible RNNs---RNNs for
which the hidden-to-hidden transition can be reversed---offer a path to reduce
the memory requirements of training, as hidden states need not be stored and
instead can be recomputed during backpropagation. We first show that perfectly
reversible RNNs, which require no storage of the hidden activations, are
fundamentally limited because they cannot forget information from their hidden
state. We then provide a scheme for storing a small number of bits in order to
allow perfect reversal with forgetting. Our method achieves comparable
performance to traditional models while reducing the activation memory cost by
a factor of 10--15. We extend our technique to attention-based
sequence-to-sequence models, where it maintains performance while reducing
activation memory cost by a factor of 5--10 in the encoder, and a factor of
10--15 in the decoder.

该研究提出了一种称为可逆 RNN 的方法，用于优化循环神经网络，该方法可以在不存储 hidden activations 的情况下实现完全可逆，并降低激活内存成本，扩展到 attention-based sequence-to-sequence 模型中效果显著。