Transformers have emerged as the backbone of large language models (LLMs).
However, generation remains inefficient due to the need to store in memory a
cache of key-value representations for past tokens, whose size scales linearly
with the input sequence length and batch size. As a solution, we propose
Dynamic Memory Compression (DMC), a method for on-line key-value cache
compression at inference time. Most importantly, the model learns to apply
different compression rates in different heads and layers. We retrofit
pre-trained LLMs such as Llama 2 (7B, 13B and 70B) into DMC Transformers,
achieving up to ~3.7x throughput increase in auto-regressive inference on a
NVIDIA H100 GPU. DMC is applied via continued pre-training on a negligible
percentage of the original data without adding any extra parameters. We find
that DMC preserves the original downstream performance with up to 4x cache
compression, outperforming up-trained grouped-query attention (GQA). GQA and
DMC can be even combined to obtain compounded gains. As a result DMC fits
longer contexts and larger batches within any given memory budget.

通过动态内存压缩 (DMC) 方法，我们可以提高基于 Transformers 的大型语言模型 (LLMs) 在自回归推理中的吞吐量，在保持原有性能的同时，使用不同的压缩率来适应不同的注意头和层级，并可以与其他技术相结合以提供更好的结果。