Statistical language modeling and translation with transformers have found
many successful applications in program understanding and generation tasks,
setting high benchmarks for tools in modern software development environments.
The finite context window of these neural models means, however, that they will
be unable to leverage the entire relevant context of large files and packages
for any given task. While there are many efforts to extend the context window,
we introduce an architecture-independent approach for leveraging the syntactic
hierarchies of source code for incorporating entire file-level context into a
fixed-length window. Using concrete syntax trees of each source file we extract
syntactic hierarchies and integrate them into context window by selectively
removing from view more specific, less relevant scopes for a given task. We
evaluate this approach on code generation tasks and joint translation of
natural language and source code in Python programming language, achieving a
new state-of-the-art in code completion and summarization for Python in the
CodeXGLUE benchmark. We also introduce new CodeXGLUE benchmarks for
user-experience-motivated tasks: code completion with normalized literals,
method body completion/code summarization conditioned on file-level context.

通过使用语法树，将源代码的整个文件级上下文集成到一个固定长度的窗口中，从而实现了为编程任务提供上下文而不丧失大文件和包的整个相关上下文的目标，表现出了优秀的性能表现。

eWASH: 通过语法层次结构扩展窗口访问，对源代码文件进行长距离建模

Long-Range Modeling of Source Code Files with eWASH: Extended Window  Access by Syntax Hierarchy

How and to what extent does BERT encode syntactically-sensitive hierarchical
information or positionally-sensitive linear information? Recent work has shown
that contextual representations like BERT perform well on tasks that require
sensitivity to linguistic structure. We present here two studies which aim to
provide a better understanding of the nature of BERT's representations. The
first of these focuses on the identification of structurally-defined elements
using diagnostic classifiers, while the second explores BERT's representation
of subject-verb agreement and anaphor-antecedent dependencies through a
quantitative assessment of self-attention vectors. In both cases, we find that
BERT encodes positional information about word tokens well on its lower layers,
but switches to a hierarchically-oriented encoding on higher layers. We
conclude then that BERT's representations do indeed model linguistically
relevant aspects of hierarchical structure, though they do not appear to show
the sharp sensitivity to hierarchical structure that is found in human
processing of reflexive anaphora.

本文探讨了 BERT 对句法层次结构、位置信息以及自注意力向量的编码，并发现 BERT 在较低的层次上良好地编码了位置信息，在较高的层次上则更倾向于编码层次结构，这表明 BERT 的表示法确实模拟了语言的一些层次结构，并且对于处理反身代词的普遍敏感性没有人类大脑那么强。