Large Language Models (LLMs) have demonstrated impressive performance in
various NLP tasks, but they still suffer from challenges such as hallucination
and weak numerical reasoning. To overcome these challenges, external tools can
be used to enhance LLMs' question-answering abilities. However, current
evaluation methods do not distinguish between questions that can be answered
using LLMs' internal knowledge and those that require external information
through tool use. To address this issue, we introduce a new dataset called
ToolQA, which is designed to faithfully evaluate LLMs' ability to use external
tools for question answering. Our development of ToolQA involved a scalable,
automated process for dataset curation, along with 13 specialized tools
designed for interaction with external knowledge in order to answer questions.
Importantly, we strive to minimize the overlap between our benchmark data and
LLMs' pre-training data, enabling a more precise evaluation of LLMs' tool-use
reasoning abilities. We conducted an in-depth diagnosis of existing tool-use
LLMs to highlight their strengths, weaknesses, and potential improvements. Our
findings set a new benchmark for evaluating LLMs and suggest new directions for
future advancements. Our data and code are freely available to the broader
scientific community on GitHub.

为了有效评估 Large Language Models（LLMs） 使用外部工具回答问题的能力，我们开发了一个名为 ToolQA 的新数据集，并使用可伸缩的自动化过程进行数据集的管理，并使用 13 种专门设计的工具进行交互以回答问题。