Knowledge-based question answering (KBQA) is widely used in many scenarios
that necessitate domain knowledge. Large language models (LLMs) bring
opportunities to KBQA, while their costs are significantly higher and absence
of domain-specific knowledge during pre-training. We are motivated to combine
LLMs and prior small models on knowledge graphs (KGMs) for both inferential
accuracy and cost saving. However, it remains challenging since accuracy and
cost are not readily combined in the optimization as two distinct metrics. It
is also laborious for model selection since different models excel in diverse
knowledge. To this end, we propose Coke, a novel cost-efficient strategy for
KBQA with LLMs, modeled as a tailored multi-armed bandit problem to minimize
calls to LLMs within limited budgets. We first formulate the accuracy
expectation with a cluster-level Thompson Sampling for either KGMs or LLMs. A
context-aware policy is optimized to further distinguish the expert model
subject to the question semantics. The overall decision is bounded by the cost
regret according to historical expenditure on failures. Extensive experiments
showcase the superior performance of Coke, which moves the Pareto frontier with
up to 20.89% saving of GPT-4 fees while achieving a 2.74% higher accuracy on
the benchmark datasets.

Coke 是一种新颖的成本有效的 KBQA 策略，它通过将 LLMs 和 KGMs 相结合，作为一个定制的多臂赌博问题来最小化对 LLMs 的调用，从而在有限的预算内实现成本节约并提高准确性。

基于知识的大型语言模型的高效问答系统

Cost-efficient Knowledge-based Question Answering with Large Language  Models

We survey the large language model (LLM) serving area to understand the
intricate dynamics between cost-efficiency and accuracy, which is magnified by
the growing need for longer contextual understanding when deploying models at a
massive scale. Our findings reveal that works in this space optimize along
three distinct but conflicting goals: improving serving context length (C),
improving serving accuracy (A), and improving serving performance (P). Drawing
inspiration from the CAP theorem in databases, we propose a CAP principle for
LLM serving, which suggests that any optimization can improve at most two of
these three goals simultaneously. Our survey categorizes existing works within
this framework. We find the definition and continuity of user-perceived
measurement metrics are crucial in determining whether a goal has been met,
akin to prior CAP databases in the wild. We recognize the CAP principle for LLM
serving as a guiding principle, rather than a formal theorem, to inform
designers of the inherent and dynamic trade-offs in serving models. As serving
accuracy and performance have been extensively studied, this survey focuses on
works that extend serving context length and address the resulting challenges.

我们对大型语言模型（LLM）的服务领域进行了调查，以了解成本效益和准确性之间的复杂动态关系，我们发现，在这个领域中，工作以改进服务上下文长度（C）、改进服务准确性（A）和改进服务性能（P）为三个不同但相互冲突的目标进行优化。我们从数据库中的 CAP 原理汲取灵感，提出了一个适用于 LLM 服务的 CAP 原则，该原则表明任何优化最多只能同时提高这三个目标中的两个。我们对现有的工作进行了分类，在这个框架内进行了调查。我们发现，用户感知的度量指标的定义和连续性对于确定目标是否达到至关重要，类似于以往的现实中的 CAP 数据库。我们将 LLM 服务的 CAP 原则作为指导原则，而不是正式定理，以通知设计师在服务模型时的固有和动态的权衡。鉴于服务准确性和性能已经得到广泛研究，本调查重点关注扩展服务上下文长度和应对由此产生的挑战的工作。

LLM 服务的 CAP 原则

The CAP Principle for LLM Serving

Training deep learning models in the cloud or on dedicated hardware is
expensive. A more cost-efficient option are hyperscale clouds offering spot
instances, a cheap but ephemeral alternative to on-demand resources. As spot
instance availability can change depending on the time of day, continent, and
cloud provider, it could be more cost-efficient to distribute resources over
the world. Still, it has not been investigated whether geo-distributed,
data-parallel spot deep learning training could be a more cost-efficient
alternative to centralized training.
This paper aims to answer the question: Can deep learning models be
cost-efficiently trained on a global market of spot VMs spanning different data
centers and cloud providers? To provide guidance, we extensively evaluate the
cost and throughput implications of training in different zones, continents,
and clouds for representative CV and NLP models. To expand the current training
options further, we compare the scalability potential for hybrid-cloud
scenarios by adding cloud resources to on-premise hardware to improve training
throughput. Finally, we show how leveraging spot instance pricing enables a new
cost-efficient way to train models with multiple cheap VMs, trumping both more
centralized and powerful hardware and even on-demand cloud offerings at
competitive prices.

本篇论文旨在探讨通过使用分布在不同数据中心和云提供商中的 Spot VMs 在全球市场上更加高效地培训深度学习模型的可行性及其与集中式训练的成本和吞吐量的比较，并揭示了通过利用 spot instance 定价来培训模型的一种新的成本高效方法，该方法胜过更集中和更强大的硬件，甚至优于竞争价格和按需云服务。