This study introduces a new long-form database question answering dataset
designed to evaluate how Large Language Models (LLMs) interact with a SQL
interpreter. The task necessitates LLMs to strategically generate multiple SQL
queries to retrieve sufficient data from a database, to reason with the
acquired context, and to synthesize them into a comprehensive analytical
narrative. Our findings highlight that this task poses great challenges even
for the state-of-the-art GPT-4 model. We propose and evaluate two interaction
strategies, and provide a fine-grained analysis of the individual stages within
the interaction. A key discovery is the identification of two primary
bottlenecks hindering effective interaction: the capacity for planning and the
ability to generate multiple SQL queries. To address the challenge of
accurately assessing answer quality, we introduce a multi-agent evaluation
framework that simulates the academic peer-review process, enhancing the
precision and reliability of our evaluations. This framework allows for a more
nuanced understanding of the strengths and limitations of current LLMs in
complex retrieval and reasoning tasks.

本研究介绍了一个新的长格式数据库问答数据集，旨在评估大型语言模型（LLMs）与 SQL 解释器的互动。研究发现即使对于最先进的 GPT-4 模型，这个任务也存在巨大挑战。我们提出并评估了两种互动策略，并对互动过程中的各个阶段进行了细致分析。一个重要发现是确定了两个主要瓶颈，即规划能力和生成多个 SQL 查询能力。为了解决准确评估答案质量的挑战，我们引入了一个多代理评估框架，模拟学术同行评审过程，增强了我们评估的精确性和可靠性。该框架使我们能够更加细致地了解当前 LLMs 在复杂检索和推理任务中的优点和局限性。