Text-to-SQL, which involves translating natural language into Structured Query Language (SQL), is crucial for enabling broad access to structured databases without expert knowledge. However, designing models for such tasks is challenging due to numerous factors, including the presence of 'noise,' such as ambiguous questions and syntactical errors. This study provides an in-depth analysis of the distribution and types of noise in the widely used BIRD-Bench benchmark and the impact of noise on models. While BIRD-Bench was created to model dirty and noisy database values, it was not created to contain noise and errors in the questions and gold queries. We found that noise in questions and gold queries are prevalent in the dataset, with varying amounts across domains, and with an uneven distribution between noise types. The presence of incorrect gold SQL queries, which then generate incorrect gold answers, has a significant impact on the benchmark's reliability. Surprisingly, when evaluating models on corrected SQL queries, zero-shot baselines surpassed the performance of state-of-the-art prompting methods. We conclude that informative noise labels and reliable benchmarks are crucial to developing new Text-to-SQL methods that can handle varying types of noise.

文本到结构化查询语言（SQL）的转换对于使结构化数据库可以广泛访问而无需专业知识至关重要。这项研究深入分析了广泛使用的BIRD-Bench基准测试中噪声的分布和类型，以及噪声对模型的影响。我们发现问题和黄金查询中的噪声在数据集中普遍存在，域之间的数量存在差异，并且噪音类型分布不均。不正确的黄金SQL查询会生成不正确的黄金答案，对基准测试的可靠性有重大影响。令人惊讶的是，在纠正后的SQL查询上，零翻译基准超过了最先进的提示方法的性能。我们得出结论认为，信息噪声标签和可靠的基准测试对于开发能处理不同类型噪声的新文本到SQL方法至关重要。

文本转SQL中噪音效应的理解：BIRD-Bench基准测试的研究