Sparse attention as a efficient method can significantly decrease the
computation cost, but current sparse attention tend to rely on window self
attention which block the global information flow. For this problem, we present
Shifted Cross Chunk Attention (SCCA), using different KV shifting strategy to
extend respective field in each attention layer. Except, we combine Dilated
Attention(DA) and Dilated Neighborhood Attention(DNA) to present Shifted
Dilated Attention(SDA). Both SCCA and SDA can accumulate attention results in
multi head attention to obtain approximate respective field in full attention.
In this paper, we conduct language modeling experiments using different pattern
of SCCA and combination of SCCA and SDA. The proposed shifted cross chunk
attention (SCCA) can effectively extend large language models (LLMs) to longer
context combined with Positional interpolation(PI) and LoRA than current sparse
attention. Notably, SCCA adopts LLaMA2 7B from 4k context to 8k in single V100.
This attention pattern can provide a Plug-and-play fine-tuning method to extend
model context while retaining their original architectures, and is compatible
with most existing techniques.

通过 Shifted Cross Chunk Attention (SCCA) 结合 Dilated Attention (DA) 和 Dilated Neighborhood Attention (DNA) 的方法，本研究在语言建模实验中使用不同的 SCCA 模式和 SCCA 与 SDA 的组合，有效地将大型语言模型（LLMs）扩展到更长的上下文，并与当前稀疏注意力相比提供了一个插拔式微调方法，以扩大模型上下文，并与大多数现有技术兼容。