A code generation system generates programming language code based on an input natural language description. State-of-the-art approaches rely on neural networks for code generation. However, these code generators suffer from two problems. One is the long dependency problem, where a code element often depends on another far-away code element. A variable reference, for example, depends on its definition, which may appear quite a few lines before. The other problem is structure modeling, as programs contain rich structural information. In this paper, we propose a novel tree-based neural architecture, TreeGen, for code generation. TreeGen uses the attention mechanism of Transformers to alleviate the long-dependency problem, and introduces a novel AST reader (encoder) to incorporate grammar rules and AST structures into the network. We evaluated TreeGen on a Python benchmark, HearthStone, and two semantic parsing benchmarks, ATIS and GEO. TreeGen outperformed the previous state-of-the-art approach by 4.5 percentage points on HearthStone, and achieved the best accuracy among neural network-based approaches on ATIS (89.1%) and GEO (89.6%). We also conducted an ablation test to better understand each component of our model.

本文提出了一种基于树结构的神经架构(TreeGen)来生成编程语言代码，该模型使用了Transformer的注意力机制解决了长依赖问题，并引入了一种新颖的AST阅读器(encoder)将语法规则和AST结构整合到网络中。在Python基准测试、ATIS和GEO语义解析基准测试中，TreeGen的性能优于之前的最先进方法4.5个百分点，同时在ATIS（89.1%）和GEO（89.6%）的神经网络方法中实现了最佳精度。

TreeGen: 基于树状结构的Transformer编码生成架构