Recent studies on software tool manipulation with large language models
(LLMs) mostly rely on closed model APIs. The industrial adoption of these
models is substantially constrained due to the security and robustness risks in
exposing information to closed LLM API services. In this paper, we ask can we
enhance open-source LLMs to be competitive to leading closed LLM APIs in tool
manipulation, with practical amount of human supervision. By analyzing common
tool manipulation failures, we first demonstrate that open-source LLMs may
require training with usage examples, in-context demonstration and generation
style regulation to resolve failures. These insights motivate us to revisit
classical methods in LLM literature, and demonstrate that we can adapt them as
model alignment with programmatic data generation, system prompts and
in-context demonstration retrievers to enhance open-source LLMs for tool
manipulation. To evaluate these techniques, we create the ToolBench, a tool
manipulation benchmark consisting of diverse software tools for real-world
tasks. We demonstrate that our techniques can boost leading open-source LLMs by
up to 90% success rate, showing capabilities competitive to OpenAI GPT-4 in 4
out of 8 ToolBench tasks. We show that such enhancement typically requires
about one developer day to curate data for each tool, rendering a recipe with
practical amount of human supervision.

研究了在开源 LLM 中通过一定程度的人工监督和对数据进行生成和调整，将其在工具操作方面提升至与已有的闭源 LLM 相竞争的能力，同时设计了一个工具操纵基准进行了实验，结果表明，在大多数情况下，该方法可使其成功率提高至 90％。