Large Language Models (LLMs) have become increasingly popular, transforming a
wide range of applications across various domains. However, the real-world
effectiveness of their query cache systems has not been thoroughly
investigated. In this work, we for the first time conducted an analysis on
real-world human-to-LLM interaction data, identifying key challenges in
existing caching solutions for LLM-based chat services. Our findings reveal
that current caching methods fail to leverage semantic connections, leading to
inefficient cache performance and extra token costs. To address these issues,
we propose SCALM, a new cache architecture that emphasizes semantic analysis
and identifies significant cache entries and patterns. We also detail the
implementations of the corresponding cache storage and eviction strategies. Our
evaluations show that SCALM increases cache hit ratios and reduces operational
costs for LLMChat services. Compared with other state-of-the-art solutions in
GPTCache, SCALM shows, on average, a relative increase of 63% in cache hit
ratio and a relative improvement of 77% in tokens savings.

通过对真实世界的人机交互数据进行分析，我们首次对基于大型语言模型的聊天服务的现有缓存解决方案进行了分析，发现当前的缓存方法未能充分利用语义连接，导致缓存性能低效和额外的标记成本。为了解决这些问题，我们提出了一种新的缓存架构 SCALM，强调语义分析，并识别重要的缓存条目和模式。我们还详细介绍了相应的缓存存储和清除策略的实现。评估结果显示，SCALM 提高了 LLMChat 服务的缓存命中率，并降低了运行成本。与 GPTCache 等其他最先进的解决方案相比，SCALM 的平均缓存命中率相对提高了 63％，标记节省率相对提高了 77％。

面向大型语言模型自动聊天服务的语义缓存

SCALM: Towards Semantic Caching for Automated Chat Services with Large  Language Models

This paper addresses the escalating challenge of redundant data transmission
in networks. The surge in traffic has strained backhaul links and backbone
networks, prompting the exploration of caching solutions at the edge router.
Existing work primarily relies on Markov Decision Processes (MDP) for caching
issues, assuming fixed-time interval decisions; however, real-world scenarios
involve random request arrivals, and despite the critical role of various file
characteristics in determining an optimal caching policy, none of the related
existing work considers all these file characteristics in forming a caching
policy. In this paper, first, we formulate the caching problem using a
semi-Markov Decision Process (SMDP) to accommodate the continuous-time nature
of real-world scenarios allowing for caching decisions at random times upon
file requests. Then, we propose a double deep Q-learning-based caching approach
that comprehensively accounts for file features such as lifetime, size, and
importance. Simulation results demonstrate the superior performance of our
approach compared to a recent Deep Reinforcement Learning-based method.
Furthermore, we extend our work to include a Transfer Learning (TL) approach to
account for changes in file request rates in the SMDP framework. The proposed
TL approach exhibits fast convergence, even in scenarios with increased
differences in request rates between source and target domains, presenting a
promising solution to the dynamic challenges of caching in real-world
environments.

本文探讨了网络中冗余数据传输的日益严峻挑战，提出了基于双深度强化学习缓存的解决方案，通过综合考虑文件的生命周期、大小和重要性等特征，实现了比最近的基于深度强化学习的方法更优异的性能，并引入了迁移学习以解决实际环境中缓存的动态挑战。

基于深度强化学习和迁移学习的边缘缓存

Edge Caching Based on Deep Reinforcement Learning and Transfer Learning

This paper has the following ambitious goal: to convince the reader that
content caching is an exciting research topic for the future communication
systems and networks. Caching has been studied for more than 40 years, and has
recently received increased attention from industry and academia. Novel caching
techniques promise to push the network performance to unprecedented limits, but
also pose significant technical challenges. This tutorial provides a brief
overview of existing caching solutions, discusses seminal papers that open new
directions in caching, and presents the contributions of this Special Issue. We
analyze the challenges that caching needs to address today, considering also an
industry perspective, and identify bottleneck issues that must be resolved to
unleash the full potential of this promising technique.

本研究论文旨在说明内容缓存技术对未来通信系统和网络的重要性，分析现有内容缓存技术的方法，讨论开创了新的缓存方向的论文，探讨缓存需要解决的挑战，并针对产业需求提出解决方案，以期发掘这一技术的全部潜力。