Recently, there has been significant progress in studying neural networks for
translating text descriptions into SQL queries under the zero-shot cross-domain
setting. Despite achieving good performance on some public benchmarks, we
observe that existing text-to-SQL models do not generalize when facing domain
knowledge that does not frequently appear in the training data, which may
render the worse prediction performance for unseen domains. In this work, we
investigate the robustness of text-to-SQL models when the questions require
rarely observed domain knowledge. In particular, we define five types of domain
knowledge and introduce Spider-DK (DK is the abbreviation of domain knowledge),
a human-curated dataset based on the Spider benchmark for text-to-SQL
translation. NL questions in Spider-DK are selected from Spider, and we modify
some samples by adding domain knowledge that reflects real-world question
paraphrases. We demonstrate that the prediction accuracy dramatically drops on
samples that require such domain knowledge, even if the domain knowledge
appears in the training set, and the model provides the correct predictions for
related training samples.

在研究神经网络将文本描述翻译成 SQL 查询方面，在零射跨域设置下取得了显著进展，但是现有的文本到 SQL 模型在面对训练数据中很少出现的领域知识时不具有普适性。本文介绍了一种人工筛选的数据集 Spider-DK，用于研究文本到 SQL 模型在需要应用很少出现的领域知识时的预测表现。在 Spider-DK 中，我们通过添加反映现实世界问题的领域知识来修改一些 Spider 的样本，并证明当样本需要这种领域知识时，预测准确率显著下降，即使该领域知识出现在训练集中，模型对相关训练样本的预测仍然正确。