We present Prompt Cache, an approach for accelerating inference for large
language models (LLM) by reusing attention states across different LLM prompts.
Many input prompts have overlapping text segments, such as system messages,
prompt templates, and documents provided for context. Our key insight is that
by precomputing and storing the attention states of these frequently occurring
text segments on the inference server, we can efficiently reuse them when these
segments appear in user prompts. Prompt Cache employs a schema to explicitly
define such reusable text segments, called prompt modules. The schema ensures
positional accuracy during attention state reuse and provides users with an
interface to access cached states in their prompt. Using a prototype
implementation, we evaluate Prompt Cache across several LLMs. We show that
Prompt Cache significantly reduce latency in time-to-first-token, especially
for longer prompts such as document-based question answering and
recommendations. The improvements range from 8x for GPU-based inference to 60x
for CPU-based inference, all while maintaining output accuracy and without the
need for model parameter modifications.

使用 Prompt Cache 方法，可以通过在不同的大型语言模型提示之间重复使用注意力状态来加快推理速度。这种方法通过预先计算和存储输入提示中经常出现的文本段的注意力状态，以在用户提示中高效地重用它们。在多个大型语言模型上的评估显示，Prompt Cache 显著减少了从第一个标记到输出的延迟，尤其对于基于文档的问答和推荐等较长的提示。改进范围从基于 GPU 的推理中的 8 倍到基于 CPU 的推理中的 60 倍，同时保持输出准确性，无需修改模型参数。