The recent trend of using Large Language Models (LLMs) as intelligent agents
in real-world applications underscores the necessity for comprehensive
evaluations of their capabilities, particularly in complex scenarios involving
planning, creating, and using tools. However, existing benchmarks typically
focus on simple synthesized queries that do not reflect real-world complexity,
thereby offering limited perspectives in evaluating tool utilization. To
address this issue, we present UltraTool, a novel benchmark designed to improve
and evaluate LLMs' ability in tool utilization within real-world scenarios.
UltraTool focuses on the entire process of using tools - from planning and
creating to applying them in complex tasks. It emphasizes real-world
complexities, demanding accurate, multi-step planning for effective
problem-solving. A key feature of UltraTool is its independent evaluation of
planning with natural language, which happens before tool usage and simplifies
the task solving by mapping out the intermediate steps. Thus, unlike previous
work, it eliminates the restriction of pre-defined toolset during planning.
Through extensive experiments on various LLMs, we offer novel insights into the
evaluation of capabilities of LLMs in tool utilization, thereby contributing a
fresh perspective to this rapidly evolving field. The benchmark is publicly
available at this https URL

通过 UltraTool，我们提供了一个新的基准来改进和评估大型语言模型在实际场景中利用工具的能力，重点关注从规划、创建到应用工具的整个过程，并强调了真实世界的复杂性，需要准确的多步规划来解决问题，从而为这个快速发展的领域提供了新的见解。