Large language models (LLMs) have achieved huge success in numerous natural
language process (NLP) tasks. However, it faces the challenge of significant
resource consumption during inference. In this paper, we aim to improve the
inference efficiency of LLMs by prompt caching, i.e., if the current prompt can
be answered by the same response of a previous prompt, one can directly utilize
that previous response without calling the LLM. Specifically, we focus on the
prediction accuracy of prompt caching for single-round question-answering tasks
via embedding similarity. The existing embeddings of prompts mostly focus on
whether two prompts are semantically similar, which is not necessarily
equivalent to whether the same response can answer them. Therefore, we propose
a distillation-based method to fine-tune the existing embeddings for better
caching prediction. Theoretically, we provide finite-sample guarantees for the
convergence of our method under different types of loss functions. Empirically,
we carefully construct a hard dataset based on Kwiatkowski et al. (2019) where
the existing embedding model (Wang et al., 2022) only achieves an AUC of 0.51.
We then fine-tune the above embedding model, which significantly improves the
AUC of caching prediction from 0.51 to 0.81. We also conduct simulations
demonstrating that our trained models achieve better caching efficiency than
the previous embedding model.

通过提示缓存改善大型语言模型的推理效率，专注于通过嵌入相似性预测单轮问答任务的提示缓存的准确性，并提出了基于蒸馏的方法来优化嵌入，实验结果显示我们的模型在缓存效率上优于之前的嵌入模型。