Transformer-based LLMs have achieved exceptional performance across a wide range of NLP tasks. However, the standard self-attention mechanism suffers from quadratic time complexity and linearly increased cache size. Sliding window attention (SWA) solves this problem by restricting the attention range to a fixed-size local context window. Nevertheless, SWA employs a uniform window size for each head in each layer, making it inefficient in capturing context of varying scales. To mitigate this limitation, we propose Multi-Scale Window Attention (MSWA) which applies diverse window sizes across heads and layers in the Transformer. It not only allows for different window sizes among heads within the same layer but also progressively increases window size allocation from shallow to deep layers, thus enabling the model to capture contextual information with different lengths and distances. Experimental results on language modeling and common-sense reasoning tasks substantiate that MSWA outperforms traditional local attention in both effectiveness and efficiency.

本研究针对传统自注意力机制在计算复杂度和缓存大小方面的限制，提出了多尺度窗口注意力（MSWA），通过在Transformer中为不同头和层采用多样化的窗口大小，以提高对变化尺度上下文的捕捉能力。实验结果表明，MSWA在语言建模和常识推理任务中在效果和效率上均优于传统局部注意力机制。

MSWA：通过多尺度窗口注意力优化局部注意力