Creating programs to correctly manipulate data is a difficult task, as the
underlying programming languages and APIs can be challenging to learn for many
users who are not skilled programmers. Large language models (LLMs) demonstrate
remarkable potential for generating code from natural language, but in the data
manipulation domain, apart from the natural language (NL) description of the
intended task, we also have the dataset on which the task is to be performed,
or the "data context". Existing approaches have utilized data context in a
limited way by simply adding relevant information from the input data into the
prompts sent to the LLM.
In this work, we utilize the available input data to execute the candidate
programs generated by the LLMs and gather their outputs. We introduce semantic
reranking, a technique to rerank the programs generated by LLMs based on three
signals coming the program outputs: (a) semantic filtering and well-formedness
based score tuning: do programs even generate well-formed outputs, (b) semantic
interleaving: how do the outputs from different candidates compare to each
other, and (c) output-based score tuning: how do the outputs compare to outputs
predicted for the same task. We provide theoretical justification for semantic
interleaving. We also introduce temperature mixing, where we combine samples
generated by LLMs using both high and low temperatures. We extensively evaluate
our approach in three domains, namely databases (SQL), data science (Pandas)
and business intelligence (Excel's Power Query M) on a variety of new and
existing benchmarks. We observe substantial gains across domains, with
improvements of up to 45% in top-1 accuracy and 34% in top-3 accuracy.

本文提出了一种利用输入数据对候选程序程序进行语义重新排序，并结合高低温度混合的技术，能够有效提升在数据操作领域中使用大型语言模型生成正确程序的准确性。