We present ControlLLM, a novel framework that enables large language models
(LLMs) to utilize multi-modal tools for solving complex real-world tasks.
Despite the remarkable performance of LLMs, they still struggle with tool
invocation due to ambiguous user prompts, inaccurate tool selection and
parameterization, and inefficient tool scheduling. To overcome these
challenges, our framework comprises three key components: (1) a \textit{task
decomposer} that breaks down a complex task into clear subtasks with
well-defined inputs and outputs; (2) a \textit{Thoughts-on-Graph (ToG)
paradigm} that searches the optimal solution path on a pre-built tool graph,
which specifies the parameter and dependency relations among different tools;
and (3) an \textit{execution engine with a rich toolbox} that interprets the
solution path and runs the tools efficiently on different computational
devices. We evaluate our framework on diverse tasks involving image, audio, and
video processing, demonstrating its superior accuracy, efficiency, and
versatility compared to existing methods.

我们介绍了 ControlLLM，这是一个新颖的框架，使得大型语言模型能够利用多模态工具来解决复杂的现实世界任务，通过使用任务分解器、Thoughts-on-Graph（ToG）范式和具备丰富工具箱的执行引擎，在图像、音频和视频处理等多样任务中展示出了超群的准确性、效率和多功能性。