The self-attention mechanism in the transformer architecture is capable of
capturing long-range dependencies and it is the main reason behind its
effectiveness in processing sequential data. Nevertheless, despite their
success, transformers have two significant drawbacks that still limit their
broader applicability: (1) In order to remember past information, the
self-attention mechanism requires access to the whole history to be provided as
context. (2) The inference cost in transformers is expensive. In this paper we
introduce recurrent alternatives to the transformer self-attention mechanism
that offer a context-independent inference cost, leverage long-range
dependencies effectively, and perform well in practice. We evaluate our
approaches in reinforcement learning problems where the aforementioned
computational limitations make the application of transformers nearly
infeasible. We quantify the impact of the different components of our
architecture in a diagnostic environment and assess performance gains in 2D and
3D pixel-based partially-observable environments. When compared to a
state-of-the-art architecture, GTrXL, inference in our approach is at least 40%
cheaper while reducing memory use in more than 50%. Our approach either
performs similarly or better than GTrXL, improving more than 37% upon GTrXL
performance on harder tasks.

通过引入循环替代方案以解决 transformer 自注意机制中的两个局限，本文提出了一种能够以较低成本进行推理并有效利用长程依赖的 transformer 自注意机制替代方法，在强化学习问题中实现了性能的提升。