Large pre-trained language models have been used to generate code,providing a flexible interface for synthesizing programs from natural language specifications. However, they often violate syntactic and semantic rules of their output language, limiting their practical usability. In this paper, we propose Synchromesh: a framework for substantially improving the reliability of pre-trained models for code generation. Synchromesh comprises two components. First, it retrieves few-shot examples from a training bank using Target Similarity Tuning (TST), a novel method for semantic example selection. TST learns to recognize utterances that describe similar target programs despite differences in surface natural language features. Then, Synchromesh feeds the examples to a pre-trained language model and samples programs using Constrained Semantic Decoding (CSD): a general framework for constraining the output to a set of valid programs in the target language. CSD leverages constraints on partial outputs to sample complete correct programs, and needs neither re-training nor fine-tuning of the language model. We evaluate our methods by synthesizing code from natural language descriptions using GPT-3 and Codex in three real-world languages: SQL queries, Vega-Lite visualizations and SMCalFlow programs. These domains showcase rich constraints that CSD is able to enforce, including syntax, scope, typing rules, and contextual logic. We observe substantial complementary gains from CSD and TST in prediction accuracy and in effectively preventing run-time errors.

本文提出了Synchromesh作为一种增强预训练语言模型在代码生成中可靠性的框架。它通过Target Similarity Tuning选择语义上相似的训练样例，并采用Constrained Semantic Decoding方法在不需要额外训练的前提下约束输出代码的合法性，从而提高了模型的实用性和运行效率。作者在使用GPT-3和Codex两种模型在SQL查询、Vega-Lite可视化和SMCalFlow编程语言中进行实验，展示了CSD在约束语义、范围、类型规则和上下文逻辑等方面的有效性。

Synchromesh：基于预训练语言模型的可靠代码生成