Neural text-to-SQL models have achieved remarkable performance in translating
natural language questions into SQL queries. However, recent studies reveal
that text-to-SQL models are vulnerable to task-specific perturbations. Previous
curated robustness test sets usually focus on individual phenomena. In this
paper, we propose a comprehensive robustness benchmark based on Spider, a
cross-domain text-to-SQL benchmark, to diagnose the model robustness. We design
17 perturbations on databases, natural language questions, and SQL queries to
measure the robustness from different angles. In order to collect more
diversified natural question perturbations, we utilize large pretrained
language models (PLMs) to simulate human behaviors in creating natural
questions. We conduct a diagnostic study of the state-of-the-art models on the
robustness set. Experimental results reveal that even the most robust model
suffers from a 14.0% performance drop overall and a 50.7% performance drop on
the most challenging perturbation. We also present a breakdown analysis
regarding text-to-SQL model designs and provide insights for improving model
robustness.

本文提出了一个跨领域文本到 SQL 基准（Spider）的综合健壮性评测来诊断模型的鲁棒性，并设计了 17 个数据库、自然语言问题和 SQL 查询的扰动来从不同角度衡量其鲁棒性。实验结果表明，即使是最鲁棒的模型在最具挑战性的扰动上也会出现 50.7% 的性能下降，分析了文本到 SQL 模型设计并提出了改进鲁棒性的见解。

Dr.Spider：一种针对文本到 SQL 稳健性的诊断评估基准

Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness

Recent neural text-to-SQL models can effectively translate natural language
questions to corresponding SQL queries on unseen databases. Working mostly on
the Spider dataset, researchers have proposed increasingly sophisticated
solutions to the problem. Contrary to this trend, in this paper we focus on
simplifications. We begin by building DuoRAT, a re-implementation of the
state-of-the-art RAT-SQL model that unlike RAT-SQL is using only relation-aware
or vanilla transformers as the building blocks. We perform several ablation
experiments using DuoRAT as the baseline model. Our experiments confirm the
usefulness of some techniques and point out the redundancy of others, including
structural SQL features and features that link the question with the schema.

本研究旨在建立 DuoRAT 作为对最新的 RAT-SQL 模型的重新实现，验证其在使用关系感知或普通变换器作为构建块的情况下的有效性，并通过消融实验来确认一些技术的有用性以及结构化 SQL 功能和在问题与架构之间建立联系的冗余特征。

DuoRAT: 迈向更简单的文本到 SQL 模型

DuoRAT: Towards Simpler Text-to-SQL Models

The sequence-to-sequence paradigm employed by neural text-to-SQL models
typically performs token-level decoding and does not consider generating SQL
hierarchically from a grammar. Grammar-based decoding has shown significant
improvements for other semantic parsing tasks, but SQL and other general
programming languages have complexities not present in logical formalisms that
make writing hierarchical grammars difficult. We introduce techniques to handle
these complexities, showing how to construct a schema-dependent grammar with
minimal over-generation. We analyze these techniques on ATIS and Spider, two
challenging text-to-SQL datasets, demonstrating that they yield 14--18\%
relative reductions in error.

本文提出了一种处理 neural text-to-SQL models 中编写语法的复杂性的技术，从而生成 schema-dependent grammar，对 ATIS 和 Spider 两个挑战性的 text-to-SQL 数据集进行分析，显示可以减少 14-18% 的相对误差。