Despite the recent success associated with Large Language Models~(LLMs), they
are notably cost-prohibitive to deploy in resource-constrained environments due
to their excessive memory and computational demands. In addition to model
parameters, the key-value cache is also stored in GPU memory, growing linearly
with batch size and sequence length. As a remedy, recent works have proposed
various eviction policies for maintaining the overhead of key-value cache under
a given budget. This paper embarks on the efficacy of existing eviction
policies in terms of \textit{importance score calculation} and \textit{eviction
scope construction}. We identify the deficiency of prior policies in these two
aspects and introduce RoCo, a \underline{r}\underline{o}bust \underline{c}ache
\underline{o}mission policy based on temporal attention scores and robustness
measures. Extensive experimentation spanning prefilling and auto-regressive
decoding stages validates the superiority of RoCo. Finally, we release EasyKV,
a versatile software package dedicated to user-friendly key-value constrained
generative inference. Code available at https://github.com/DRSY/EasyKV.

本文研究了现有驱逐策略在重要性评分计算和驱逐范围构建方面的不足，并提出 RoCo，一种基于时间注意力分数和鲁棒性度量的稳健缓存省略策略。通过广泛的实验验证了 RoCo 的卓越性能，并发布了 EasyKV 软件包，专门用于用户友好的键值约束生成推断。