The Transformer architecture is crucial for numerous AI models, but it still
faces challenges in long-range language modeling. Though several specific
transformer architectures have been designed to tackle issues of long-range
dependencies, existing methods like Transformer-XL are plagued by a high
percentage of ineffective memories. In this study, we present a plug-and-play
strategy, known as TRAining-free Memory Selection (TRAMS), that selects tokens
participating in attention calculation based on one simple metric. This
strategy allows us to keep tokens that are likely to have a high attention
score with the current queries and ignore the other ones. We have tested our
approach on the word-level benchmark (WikiText-103) and the character-level
benchmark (enwik8), and the results indicate an improvement without having
additional training or adding additional parameters.

提出了一种称为 TRAining-free Memory Selection (TRAMS) 的插拔式策略，通过一个简单的评价指标选择参与注意力计算的令牌，从而改善长程语言建模的挑战，无需额外训练或添加参数，通过在 word-level benchmark (WikiText-103) 和 character-level benchmark (enwik8) 上的测试取得了改进的结果。