This work introduces a new Transformer model called Cached Transformer, which
uses Gated Recurrent Cached (GRC) attention to extend the self-attention
mechanism with a differentiable memory cache of tokens. GRC attention enables
attending to both past and current tokens, increasing the receptive field of
attention and allowing for exploring long-range dependencies. By utilizing a
recurrent gating unit to continuously update the cache, our model achieves
significant advancements in \textbf{six} language and vision tasks, including
language modeling, machine translation, ListOPs, image classification, object
detection, and instance segmentation. Furthermore, our approach surpasses
previous memory-based techniques in tasks such as language modeling and
displays the ability to be applied to a broader range of situations.

引入一种名为 Cached Transformer 的新型 Transformer 模型，采用门控循环缓存（GRC）注意力扩展了自注意机制，实现可微分的令牌记忆缓存。GRC 注意力使得可以关注过去和当前的令牌，增大了注意力的感受野，可用于探索长距离的依赖关系。通过使用循环门控单元持续更新缓存，我们的模型在六种语言和视觉任务中实现了显著的进展，包括语言建模、机器翻译、ListOPs、图像分类、物体检测和实例分割。此外，我们的方法超越了先前基于记忆的技术在语言建模等任务中的性能，并展示了适用于更广泛场景的能力。