In this paper, we introduce a new task for code completion that focuses on handling long code input and propose a sparse Transformer model, called LongCoder, to address this task. LongCoder employs a sliding window mechanism for self-attention and introduces two types of globally accessible tokens - bridge tokens and memory tokens - to improve performance and efficiency. Bridge tokens are inserted throughout the input sequence to aggregate local information and facilitate global interaction, while memory tokens are included to highlight important statements that may be invoked later and need to be memorized, such as package imports and definitions of classes, functions, or structures. We conduct experiments on a newly constructed dataset that contains longer code context and the publicly available CodeXGLUE benchmark. Experimental results demonstrate that LongCoder achieves superior performance on code completion tasks compared to previous models while maintaining comparable efficiency in terms of computational resources during inference. All the codes and data are available at https://github.com/microsoft/CodeBERT.

本文介绍了一种新的代码补全任务，提出了一种称为LongCoder的稀疏Transformer模型来解决这个任务。LongCoder采用滑动窗口机制进行自我注意，并引入了两种全局可访问的标记-桥接标记和内存标记-以提高性能和效率。实验结果表明，LongCoder在维护推理时的计算资源效率方面保持相当的效率，在代码完成任务中比先前的模型性能更优秀。

LongCoder：面向代码自动补全的长距离预训练语言模型