We develop an approach for improving the trustworthiness and overall accuracy
of program synthesizers based on large language models for source code. Given a
natural language description of a programming problem, our method samples both
candidate programs as well as candidate predicates specifying how the program
should behave. We learn to analyze the agreement between programs and
predicates to judge both which program is most likely to be correct, and also
judge whether the language model is able to solve the programming problem in
the first place. This latter capacity allows favoring high precision over broad
recall: fostering trust by only proposing a program when the system is certain
that it is correct.

本文基于大型语言模型，提出了一种改进程序综合器可靠性和总体准确性的方法，通过语言模型自然语言描述求解程序问题，并且学习分析程序和断言的一致性，以判断哪个程序最有可能是正确的，从而通过提供可信的解决方案来改善可靠性

我说，你验证：朝向值得信赖的神经程序合成

I Speak, You Verify: Toward Trustworthy Neural Program Synthesis

When writing programs, people have the ability to tackle a new complex task
by decomposing it into smaller and more familiar subtasks. While it is
difficult to measure whether neural program synthesis methods have similar
capabilities, what we can measure is whether they compositionally generalize,
that is, whether a model that has been trained on the simpler subtasks is
subsequently able to solve more complex tasks. In this paper, we focus on
measuring the ability of learned program synthesizers to compositionally
generalize. We first characterize several different axes along which program
synthesis methods would be desired to generalize, e.g., length generalization,
or the ability to combine known subroutines in new ways that do not occur in
the training data. Based on this characterization, we introduce a benchmark
suite of tasks to assess these abilities based on two popular existing
datasets, SCAN and RobustFill. Finally, we make first attempts to improve the
compositional generalization ability of Transformer models along these axes
through novel attention mechanisms that draw inspiration from a human-like
decomposition strategy. Empirically, we find our modified Transformer models
generally perform better than natural baselines, but the tasks remain
challenging.

本文介绍了一种测量神经程序合成方法能否组合泛化的方法，引入了基于两个流行数据集的任务基准套件，以评估合成器在组合泛化方面的能力，最后通过新的注意机制来改进变压器模型的组合泛化能力