Despite the advancements of open-source large language models (LLMs) and
their variants, e.g., LLaMA and Vicuna, they remain significantly limited in
performing higher-level tasks, such as following human instructions to use
external tools (APIs). This is because current instruction tuning largely
focuses on basic language tasks instead of the tool-use domain. This is in
contrast to state-of-the-art (SOTA) LLMs, e.g., ChatGPT, which have
demonstrated excellent tool-use capabilities but are unfortunately closed
source. To facilitate tool-use capabilities within open-source LLMs, we
introduce ToolLLM, a general tool-use framework of data construction, model
training and evaluation. We first present ToolBench, an instruction-tuning
dataset for tool use, which is created automatically using ChatGPT.
Specifically, we collect 16,464 real-world RESTful APIs spanning 49 categories
from RapidAPI Hub, then prompt ChatGPT to generate diverse human instructions
involving these APIs, covering both single-tool and multi-tool scenarios.
Finally, we use ChatGPT to search for a valid solution path (chain of API
calls) for each instruction. To make the searching process more efficient, we
develop a novel depth-first search-based decision tree (DFSDT), enabling LLMs
to evaluate multiple reasoning traces and expand the search space. We show that
DFSDT significantly enhances the planning and reasoning capabilities of LLMs.
For efficient tool-use assessment, we develop an automatic evaluator: ToolEval.
We fine-tune LLaMA on ToolBench and obtain ToolLLaMA. Our ToolEval reveals that
ToolLLaMA demonstrates a remarkable ability to execute complex instructions and
generalize to unseen APIs, and exhibits comparable performance to ChatGPT. To
make the pipeline more practical, we devise a neural API retriever to recommend
appropriate APIs for each instruction, negating the need for manual API
selection.

通过引入 ToolLLM，一个包括数据构建、模型训练和评估的通用工具使用框架，我们展示了它在增强自然语言模型的规划和推理能力方面的显著影响。我们通过用 ChatGPT 创建一个工具使用指导数据集 ToolBench，并使用深度优先搜索决策树（DFSDT）扩展搜索空间，有效地获取有效的解决方案路径。通过对 LLaMA 进行微调后得到 ToolLLaMA，我们的评估器 ToolEval 显示 ToolLLaMA 在执行复杂指令和推广到未见过的 API 方面表现出卓越的能力，并与 ChatGPT 有着相当的性能。为了使流程更加实用，我们设计了一个神经 API 检索器以为每个指令推荐适当的 API，省去了手动选择 API 的繁琐步骤。