Text-to-SQL benchmarks play a crucial role in evaluating the progress made in
the field and the ranking of different models. However, accurately matching a
model-generated SQL query to a reference SQL query in a benchmark fails for
various reasons, such as underspecified natural language queries, inherent
assumptions in both model-generated and reference queries, and the
non-deterministic nature of SQL output under certain conditions. In this paper,
we conduct an extensive study of several prominent cross-domain text-to-SQL
benchmarks and re-evaluate some of the top-performing models within these
benchmarks, by both manually evaluating the SQL queries and rewriting them in
equivalent expressions. Our evaluation reveals that attaining a perfect
performance on these benchmarks is unfeasible due to the multiple
interpretations that can be derived from the provided samples. Furthermore, we
find that the true performance of the models is underestimated and their
relative performance changes after a re-evaluation. Most notably, our
evaluation reveals a surprising discovery: a recent GPT4-based model surpasses
the gold standard reference queries in the Spider benchmark in our human
evaluation. This finding highlights the importance of interpreting benchmark
evaluations cautiously, while also acknowledging the critical role of
additional independent evaluations in driving advancements in the field.

通过对几个重要的跨领域文本到 SQL 基准进行广泛的研究和重新评估，本文发现由于提供的样本可能有多个解释，这些基准的完美表现是不可行的，并且在重新评估后，模型的相对性能会发生变化。最令人瞩目的是，我们的评估发现，最近基于 GPT4 的模型在人工评估中超过了 Spider 基准中的金标准参考查询，这一发现强调了在谨慎解读基准评估的同时，独立评估在推动该领域进展中起到关键作用。

评估跨领域文本到 SQL 模型和基准

Evaluating Cross-Domain Text-to-SQL Models and Benchmarks

Text-to-SQL aims at generating SQL queries for the given natural language
questions and thus helping users to query databases. Prompt learning with large
language models (LLMs) has emerged as a recent approach, which designs prompts
to lead LLMs to understand the input question and generate the corresponding
SQL. However, it faces challenges with strict SQL syntax requirements. Existing
work prompts the LLMs with a list of demonstration examples (i.e. question-SQL
pairs) to generate SQL, but the fixed prompts can hardly handle the scenario
where the semantic gap between the retrieved demonstration and the input
question is large. In this paper, we propose a retrieval-augmented prompting
method for a LLM-based Text-to-SQL framework, involving sample-aware prompting
and a dynamic revision chain. Our approach incorporates sample-aware
demonstrations, which include the composition of SQL operators and fine-grained
information related to the given question. To retrieve questions sharing
similar intents with input questions, we propose two strategies for assisting
retrieval. Firstly, we leverage LLMs to simplify the original questions,
unifying the syntax and thereby clarifying the users' intentions. To generate
executable and accurate SQLs without human intervention, we design a dynamic
revision chain which iteratively adapts fine-grained feedback from the
previously generated SQL. Experimental results on three Text-to-SQL benchmarks
demonstrate the superiority of our method over strong baseline models.

本文提出一种基于大型语言模型的 Text-to-SQL 框架 —— 检索增强提示法，包括样本感知提示和动态修订链，通过引入检索来生成针对自然语言问题的 SQL 查询。