In this preliminary study, we investigate a GPT-driven intent-based reasoning
approach to streamline tool selection for large language models (LLMs) aimed at
system efficiency. By identifying the intent behind user prompts at runtime, we
narrow down the API toolset required for task execution, reducing token
consumption by up to 24.6\%. Early results on a real-world, massively parallel
Copilot platform with over 100 GPT-4-Turbo nodes show cost reductions and
potential towards improving LLM-based system efficiency.

通过识别用户提示背后的意图，我们调查了一种基于 GPT 的意图推理方法，以简化大型语言模型（LLMs）的工具选择，从而提高系统的效率。通过在运行时缩小任务执行所需的 API 工具集合，我们将记号消耗减少了多达 24.6％。在一个具有 100 个 GPT-4-Turbo 节点的实际大规模并行 Copilot 平台上的初步结果显示出成本的降低和改善 LLM 系统效率的潜力。

GeckOpt：通过基于意图的工具选择提高低水平管理系统效率

GeckOpt: LLM System Efficiency via Intent-Based Tool Selection

Large language models (LLMs) have garnered significant attention due to their
impressive natural language processing (NLP) capabilities. Recently, many
studies have focused on the tool utilization ability of LLMs. They primarily
investigated how LLMs effectively collaborate with given specific tools.
However, in scenarios where LLMs serve as intelligent agents, as seen in
applications like AutoGPT and MetaGPT, LLMs are expected to engage in intricate
decision-making processes that involve deciding whether to employ a tool and
selecting the most suitable tool(s) from a collection of available tools to
fulfill user requests. Therefore, in this paper, we introduce MetaTool, a
benchmark designed to evaluate whether LLMs have tool usage awareness and can
correctly choose tools. Specifically, we create a dataset called ToolE within
the benchmark. This dataset contains various types of user queries in the form
of prompts that trigger LLMs to use tools, including both single-tool and
multi-tool scenarios. Subsequently, we set the tasks for both tool usage
awareness and tool selection. We define four subtasks from different
perspectives in tool selection, including tool selection with similar choices,
tool selection in specific scenarios, tool selection with possible reliability
issues, and multi-tool selection. We conduct experiments involving nine popular
LLMs and find that the majority of them still struggle to effectively select
tools, highlighting the existing gaps between LLMs and genuine intelligent
agents. However, through the error analysis, we found there is still
significant room for improvement. Finally, we conclude with insights for tool
developers that follow ChatGPT to provide detailed descriptions that can
enhance the tool selection performance of LLMs.

本文提出了 MetaTool，这是一个用于评估大型语言模型（LLMs）的工具使用意识和正确选择工具能力的基准测试，并通过实验证明大多数 LLMs 在工具选择方面仍然存在困难。