This paper presents a new tool learning dataset Seal-Tools, which contains
self-instruct API-like tools. Seal-Tools not only offers a large number of
tools, but also includes instances which demonstrate the practical application
of tools. Seeking to generate data on a large scale while ensuring reliability,
we propose a self-instruct method to generate tools and instances, allowing
precise control over the process. Moreover, our Seal-Tools contains hard
instances that call multiple tools to complete the job, among which some are
nested tool callings. For precise and comprehensive evaluation, we use strict
format control and design three metrics from different dimensions. Therefore,
Seal-Tools can serve as a new benchmark to evaluate the tool-calling ability of
LLMs. Finally, we evaluate several prevalent LLMs and our finetuned model on
Seal-Tools. The results show that current systems are far from perfect. The
code, data and experiment results are available at
this https URL .

本研究提出了一个新的工具学习数据集 Seal-Tools，它包含了自学习的类似 API 的工具。通过提供大量的工具和实例，Seal-Tools 不仅能实现数据的大规模生成，还能展示工具的实际应用。为了确保可靠性，我们提出了一种自学习方法来生成工具和实例，使得过程能够被精确控制。此外，Seal-Tools 包含一些难度较高的实例，其中有些实例需要调用多个嵌套的工具来完成任务。为了进行精确和综合的评估，我们采用了严格的格式控制，并从不同角度设计了三个评估指标。因此，Seal-Tools 可以作为评估 LLMs 的工具调用能力的新基准。最后，我们在 Seal-Tools 上评估了几种流行的 LLMs 和我们微调的模型。结果表明当前的系统还远未达到完美。解码、数据和实验结果可在此链接获取。