Recent advancements in large language models (LLMs) have enabled in-context
learning (ICL)-based methods that significantly outperform fine-tuning
approaches for text-to-SQL tasks. However, their performance is still
considerably lower than that of human experts on benchmarks that include
complex schemas and queries, such as BIRD. This study considers the sensitivity
of LLMs to the prompts and introduces a novel approach that leverages multiple
prompts to explore a broader search space for possible answers and effectively
aggregate them. Specifically, we robustly refine the database schema through
schema linking using multiple prompts. Thereafter, we generate various
candidate SQL queries based on the refined schema and diverse prompts. Finally,
the candidate queries are filtered based on their confidence scores, and the
optimal query is obtained through a multiple-choice selection that is presented
to the LLM. When evaluated on the BIRD and Spider benchmarks, the proposed
method achieved execution accuracies of 65.5\% and 89.6\%, respectively,
significantly outperforming previous ICL-based methods. Moreover, we
established a new SOTA performance on the BIRD in terms of both the accuracy
and efficiency of the generated queries.

通过使用多个提示的多个方案和借助多个提示，本研究提出了一种新的方法来增强大型语言模型对复杂数据库模式和查询的感知能力，在 BIRD 和 Spider 基准测试中，该方法的执行准确度分别达到了 65.5% 和 89.6%，明显优于之前基于语境学习的方法，并在生成的查询准确度和效率方面建立了新的 SOTA 业绩。

MCS-SQL: 利用多个提示和多项选择进行文本到 SQL 生成

MCS-SQL: Leveraging Multiple Prompts and Multiple-Choice Selection For  Text-to-SQL Generation

Text-to-SQL parsing, which aims at converting natural language instructions
into executable SQLs, has gained increasing attention in recent years. In
particular, Codex and ChatGPT have shown impressive results in this task.
However, most of the prevalent benchmarks, i.e., Spider, and WikiSQL, focus on
database schema with few rows of database contents leaving the gap between
academic study and real-world applications. To mitigate this gap, we present
Bird, a big benchmark for large-scale database grounded in text-to-SQL tasks,
containing 12,751 pairs of text-to-SQL data and 95 databases with a total size
of 33.4 GB, spanning 37 professional domains. Our emphasis on database values
highlights the new challenges of dirty database contents, external knowledge
between NL questions and database contents, and SQL efficiency, particularly in
the context of massive databases. To solve these problems, text-to-SQL models
must feature database value comprehension in addition to semantic parsing. The
experimental results demonstrate the significance of database values in
generating accurate text-to-SQLs for big databases. Furthermore, even the most
effective text-to-SQL models, i.e. ChatGPT, only achieves 40.08% in execution
accuracy, which is still far from the human result of 92.96%, proving that
challenges still stand. Besides, we also provide an efficiency analysis to
offer insights into generating text-to-efficient-SQLs that are beneficial to
industries. We believe that BIRD will contribute to advancing real-world
applications of text-to-SQL research. The leaderboard and source code are
available: this https URL

本文提出了一个名为 Bird 的大规模基准数据集，旨在填补现有基准测试数据集在数据库值、外部知识和 SQL 某些方面的不足。实验结果表明，在大型数据库的文本到 SQL 转换中，数据库值具有重要意义。同时，最先进的文本到 SQL 模型 ChatGPT，转换准确度仅为 40.08%，距离人类结果 92.96% 还有巨大的差距。