Large Language Models (LLMs) like ChatGPT, Google Bard, Claude, and Llama 2
have revolutionized natural language processing and search engine dynamics.
However, these models incur exceptionally high computational costs. For
instance, GPT-3 consists of 175 billion parameters and inference on these
models also demands billions of floating-point operations. Caching is a natural
solution to reduce LLM inference costs on repeated queries. However, existing
caching methods are incapable of finding semantic similarities among LLM
queries, leading to unacceptable false hit-and-miss rates.
This paper introduces MeanCache, a semantic cache for LLMs that identifies
semantically similar queries to determine cache hit or miss. Using MeanCache,
the response to a user's semantically similar query can be retrieved from a
local cache rather than re-querying the LLM, thus reducing costs, service
provider load, and environmental impact. MeanCache leverages Federated Learning
(FL) to collaboratively train a query similarity model in a distributed manner
across numerous users without violating privacy. By placing a local cache in
each user's device and using FL, MeanCache reduces the latency and costs and
enhances model performance, resulting in lower cache false hit rates. Our
experiments, benchmarked against the GPTCache, reveal that MeanCache attains an
approximately 17% higher F-score and a 20% increase in precision during
semantic cache hit-and-miss decisions. Furthermore, MeanCache reduces the
storage requirement by 83% and accelerates semantic cache hit-and-miss
decisions by 11%, while still surpassing GPTCache.

MeanCache 是一种用于 LLM 的语义缓存，通过识别语义相似的查询来确定缓存是否命中，降低了计算成本，服务提供者负载和环境影响，同时使用联邦学习在各个用户之间进行分布式训练查询相似性模型，不违反隐私。与 GPTCache 相比，MeanCache 在语义缓存命中与未命中的决策中取得了大约 17% 的 F - 得分增加和 20% 的精确度增加，并且在减少存储需求方面有了 83% 的提升和在语义缓存命中与未命中的决策加速方面有了 11% 的提升，仍然超过了 GPTCache。