Large Language Models (LLMs) have emerged as a powerful tool in advancing the
Text-to-SQL task, significantly outperforming traditional methods.
Nevertheless, as a nascent research field, there is still no consensus on the
optimal prompt templates and design frameworks. Additionally, existing
benchmarks inadequately explore the performance of LLMs across the various
sub-tasks of the Text-to-SQL process, which hinders the assessment of LLMs'
cognitive capabilities and the optimization of LLM-based solutions.To address
the aforementioned issues, we firstly construct a new dataset designed to
mitigate the risk of overfitting in LLMs. Then we formulate five evaluation
tasks to comprehensively assess the performance of diverse methods across
various LLMs throughout the Text-to-SQL process.Our study highlights the
performance disparities among LLMs and proposes optimal in-context learning
solutions tailored to each task. These findings offer valuable insights for
enhancing the development of LLM-based Text-to-SQL systems.

通过构建新数据集和提出五个评估任务，全面评估不同方法在文本到 SQL 过程中的性能，揭示了大型语言模型之间的性能差异，并提出了针对每个任务的最佳上下文学习解决方案，为改进基于大型语言模型的文本到 SQL 系统的开发提供了有价值的见解。

大型语言模型的文本到 SQL 能力基准测试：全面评估

Benchmarking the Text-to-SQL Capability of Large Language Models: A  Comprehensive Evaluation

Large Language Models (LLMs) such as ChatGPT have demonstrated remarkable
performance across various tasks and have garnered significant attention from
both researchers and practitioners. However, in an educational context, we
still observe a performance gap in generating distractors -- i.e., plausible
yet incorrect answers -- with LLMs for multiple-choice questions (MCQs). In
this study, we propose a strategy for guiding LLMs such as ChatGPT, in
generating relevant distractors by prompting them with question items
automatically retrieved from a question bank as well-chosen in-context
examples. We evaluate our LLM-based solutions using a quantitative assessment
on an existing test set, as well as through quality annotations by human
experts, i.e., teachers. We found that on average 53% of the generated
distractors presented to the teachers were rated as high-quality, i.e.,
suitable for immediate use as is, outperforming the state-of-the-art model. We
also show the gains of our approach 1 in generating high-quality distractors by
comparing it with a zero-shot ChatGPT and a few-shot ChatGPT prompted with
static examples.

通过从题库自动检索题目并将其用作合适的上下文示例，本研究提出了一种引导 ChatGPT 等大型语言模型生成相关干扰项的策略，通过对现有测试集进行数量评估和教师质量评注，我们发现教师评价中有 53% 的生成干扰项被评为高质量，即可立即使用，优于最先进的模型，与零样本 ChatGPT 和少样本 ChatGPT 通过静态示例进行比较，显示了我们方法在生成高质量干扰项方面的优势。