Transformers have revolutionized deep learning in numerous fields, including
natural language processing, computer vision, and audio processing. Their
strength lies in their attention mechanism, which allows for the discovering of
complex input relationships. However, this mechanism's quadratic time and
memory complexity pose challenges for larger inputs. Researchers are now
investigating models like Linear Unified Nested Attention (Luna) or Memory
Augmented Transformer, which leverage external learnable memory to either
reduce the attention computation complexity down to linear, or to propagate
information between chunks in chunk-wise processing. Our findings challenge the
conventional thinking on these models, revealing that interfacing with the
memory directly through an attention operation is suboptimal, and that the
performance may be considerably improved by filtering the input signal before
communicating with memory.

通过对内存进行注意操作与通过输入信号筛选交互相对来说不够优化，可以利用 Luna 或 Memory Augmented Transformer 等模型减少注意计算复杂度或在块处理中传播信息来显著提高性能。