Text-to-SQL systems (also known as NL-to-SQL systems) have become an
increasingly popular solution for bridging the gap between user capabilities
and SQL-based data access. These systems translate user requests in natural
language to valid SQL statements for a specific database. Recent Text-to-SQL
systems have benefited from the rapid improvement of transformer-based language
models. However, while Text-to-SQL systems that incorporate such models
continuously reach new high scores on -- often synthetic -- benchmark datasets,
a systematic exploration of their robustness towards different data models in a
real-world, realistic scenario is notably missing. This paper provides the
first in-depth evaluation of the data model robustness of Text-to-SQL systems
in practice based on a multi-year international project focused on Text-to-SQL
interfaces. Our evaluation is based on a real-world deployment of FootballDB, a
system that was deployed over a 9 month period in the context of the FIFA World
Cup 2022, during which about 6K natural language questions were asked and
executed. All of our data is based on real user questions that were asked live
to the system. We manually labeled and translated a subset of these questions
for three different data models. For each data model, we explore the
performance of representative Text-to-SQL systems and language models. We
further quantify the impact of training data size, pre-, and post-processing
steps as well as language model inference time. Our comprehensive evaluation
sheds light on the design choices of real-world Text-to-SQL systems and their
impact on moving from research prototypes to real deployments. Last, we provide
a new benchmark dataset to the community, which is the first to enable the
evaluation of different data models for the same dataset and is substantially
more challenging than most previous datasets in terms of query complexity.

基于一个实际的部署案例和真实用户提问数据，本研究评估了 Text-to-SQL 系统在不同数据模型下的健壮性，并探讨了语言模型的性能以及训练数据规模、预处理和后处理步骤对系统性能的影响。此外，研究还为研究社区提供了一个新的基准数据集，可评估针对不同数据模型的查询复杂度。