As Large Language Models (LLMs) broaden their capabilities to manage
thousands of API calls, they are confronted with complex data operations across
vast datasets with significant overhead to the underlying system. In this work,
we introduce LLM-dCache to optimize data accesses by treating cache operations
as callable API functions exposed to the tool-augmented agent. We grant LLMs
the autonomy to manage cache decisions via prompting, seamlessly integrating
with existing function-calling mechanisms. Tested on an industry-scale
massively parallel platform that spans hundreds of GPT endpoints and terabytes
of imagery, our method improves Copilot times by an average of 1.24x across
various LLMs and prompting techniques.

本研究介绍了 LLM-dCache，它将缓存操作视为可调用的 API 函数，通过与现有的函数调用机制无缝集成，使得大型语言模型能够在管理数据访问时提高效率。在大规模平台上的测试表明，我们的方法在不同的 LLM 和提示技术上将 Copilot 的时间平均提升了 1.24 倍。

LLM-dCache: 使用 GPT 驱动的本地化数据缓存改善增强工具的 LLM

LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data  Caching

Modern AI models provide the key to a long-standing dream: processing
analytical queries about almost any kind of data. Until recently, it was
difficult and expensive to extract facts from company documents, data from
scientific papers, or insights from image and video corpora. Today's models can
accomplish these tasks with high accuracy. However, a programmer who wants to
answer a substantive AI-powered query must orchestrate large numbers of models,
prompts, and data operations. For even a single query, the programmer has to
make a vast number of decisions such as the choice of model, the right
inference method, the most cost-effective inference hardware, the ideal prompt
design, and so on. The optimal set of decisions can change as the query changes
and as the rapidly-evolving technical landscape shifts. In this paper we
present Palimpzest, a system that enables anyone to process AI-powered
analytical queries simply by defining them in a declarative language. The
system uses its cost optimization framework -- which explores the search space
of AI models, prompting techniques, and related foundation model optimizations
-- to implement the query with the best trade-offs between runtime, financial
cost, and output data quality. We describe the workload of AI-powered analytics
tasks, the optimization methods that Palimpzest uses, and the prototype system
itself. We evaluate Palimpzest on tasks in Legal Discovery, Real Estate Search,
and Medical Schema Matching. We show that even our simple prototype offers a
range of appealing plans, including one that is 3.3x faster, 2.9x cheaper, and
offers better data quality than the baseline method. With parallelism enabled,
Palimpzest can produce plans with up to a 90.3x speedup at 9.1x lower cost
relative to a single-threaded GPT-4 baseline, while obtaining an F1-score
within 83.5% of the baseline. These require no additional work by the user.

Palimpzest 是一个系统，它通过在一个声明性语言中定义查询来使任何人能够处理基于人工智能的分析查询。系统使用其成本优化框架来实现查询，在运行时间、财务成本和输出数据质量之间实现最佳权衡。该系统在法律发现、房地产搜索和医疗模式匹配等任务上展示了各种吸引人的计划，包括比基准方法更快 3.3 倍、更便宜 2.9 倍，并提供更好的数据质量的计划。与单线程的 GPT-4 基准相比，通过启用并行性，Palimpzest 可以以最多 90.3 倍的加速度、9.1 倍较低的成本获得 83.5% 的 F1 分数，而且不需要用户额外的工作。