Text-to-SQL parsing, which aims at converting natural language instructions
into executable SQLs, has gained increasing attention in recent years. In
particular, Codex and ChatGPT have shown impressive results in this task.
However, most of the prevalent benchmarks, i.e., Spider, and WikiSQL, focus on
database schema with few rows of database contents leaving the gap between
academic study and real-world applications. To mitigate this gap, we present
Bird, a big benchmark for large-scale database grounded in text-to-SQL tasks,
containing 12,751 pairs of text-to-SQL data and 95 databases with a total size
of 33.4 GB, spanning 37 professional domains. Our emphasis on database values
highlights the new challenges of dirty database contents, external knowledge
between NL questions and database contents, and SQL efficiency, particularly in
the context of massive databases. To solve these problems, text-to-SQL models
must feature database value comprehension in addition to semantic parsing. The
experimental results demonstrate the significance of database values in
generating accurate text-to-SQLs for big databases. Furthermore, even the most
effective text-to-SQL models, i.e. ChatGPT, only achieves 40.08% in execution
accuracy, which is still far from the human result of 92.96%, proving that
challenges still stand. Besides, we also provide an efficiency analysis to
offer insights into generating text-to-efficient-SQLs that are beneficial to
industries. We believe that BIRD will contribute to advancing real-world
applications of text-to-SQL research. The leaderboard and source code are
available: this https URL

本文提出了一个名为 Bird 的大规模基准数据集，旨在填补现有基准测试数据集在数据库值、外部知识和 SQL 某些方面的不足。实验结果表明，在大型数据库的文本到 SQL 转换中，数据库值具有重要意义。同时，最先进的文本到 SQL 模型 ChatGPT，转换准确度仅为 40.08%，距离人类结果 92.96% 还有巨大的差距。