Large Language Models (LLMs) have garnered considerable attention owing to
their remarkable capabilities, leading to an increasing number of companies
offering LLMs as services. Different LLMs achieve different performance at
different costs. A challenge for users lies in choosing the LLMs that best fit
their needs, balancing cost and performance. In this paper, we propose a
framework for addressing the cost-effective query allocation problem for LLMs.
Given a set of input queries and candidate LLMs, our framework, named OptLLM,
provides users with a range of optimal solutions to choose from, aligning with
their budget constraints and performance preferences, including options for
maximizing accuracy and minimizing cost. OptLLM predicts the performance of
candidate LLMs on each query using a multi-label classification model with
uncertainty estimation and then iteratively generates a set of non-dominated
solutions by destructing and reconstructing the current solution. To evaluate
the effectiveness of OptLLM, we conduct extensive experiments on various types
of tasks, including text classification, question answering, sentiment
analysis, reasoning, and log parsing. Our experimental results demonstrate that
OptLLM substantially reduces costs by 2.40% to 49.18% while achieving the same
accuracy as the best LLM. Compared to other multi-objective optimization
algorithms, OptLLM improves accuracy by 2.94% to 69.05% at the same cost or
saves costs by 8.79% and 95.87% while maintaining the highest attainable
accuracy.

我们提出了一个针对大型语言模型的成本效益查询分配问题的框架，名为 OptLLM，通过使用多标签分类模型进行性能预测，生成一系列优化解决方案，旨在满足用户的预算限制和性能偏好，包括最大化准确性和最小化成本。OptLLM 在各种类型的任务上进行了广泛的实验，包括文本分类、问答、情感分析、推理和日志解析，实验证明 OptLLM 在降低成本 2.40% 至 49.18% 的同时实现与最佳大型语言模型相同的准确性，相比其他多目标优化算法，OptLLM 在相同成本下提高 2.94% 至 69.05% 的准确性或节省 8.79% 至 95.87% 的成本并保持最高可达准确性。