Auto-regressive inference of transformers benefit greatly from Key-Value (KV)
caching, but can lead to major memory bottlenecks as model size, batch size,
and sequence length grow at scale. We introduce Multi-Layer Key-Value (MLKV)
sharing, a novel approach extending KV sharing across transformer layers to
reduce memory usage beyond what was possible with Multi-Query Attention (MQA)
and Grouped-Query Attention (GQA). Evaluations on various NLP benchmarks and
inference metrics using uptrained Pythia-160M variants demonstrate that MLKV
significantly reduces memory usage with minimal performance loss, reducing KV
cache size down to a factor of 6x compared to MQA. These results highlight
MLKV's potential for efficient deployment of transformer models at scale. We
provide code at this https URL

通过多层键 - 值共享方式，将键值缓存扩展到变压器层，进一步降低内存使用量，提高变压器模型在大规模场景中的效率。

MLKV：用于内存高效 Transformer 解码的多层键值头

MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer  Decoding

Transformers have emerged as the backbone of large language models (LLMs).
However, generation remains inefficient due to the need to store in memory a
cache of key-value representations for past tokens, whose size scales linearly
with the input sequence length and batch size. As a solution, we propose
Dynamic Memory Compression (DMC), a method for on-line key-value cache
compression at inference time. Most importantly, the model learns to apply
different compression rates in different heads and layers. We retrofit
pre-trained LLMs such as Llama 2 (7B, 13B and 70B) into DMC Transformers,
achieving up to ~3.7x throughput increase in auto-regressive inference on a
NVIDIA H100 GPU. DMC is applied via continued pre-training on a negligible
percentage of the original data without adding any extra parameters. We find
that DMC preserves the original downstream performance with up to 4x cache
compression, outperforming up-trained grouped-query attention (GQA). GQA and
DMC can be even combined to obtain compounded gains. As a result DMC fits
longer contexts and larger batches within any given memory budget.

通过动态内存压缩 (DMC) 方法，我们可以提高基于 Transformers 的大型语言模型 (LLMs) 在自回归推理中的吞吐量，在保持原有性能的同时，使用不同的压缩率来适应不同的注意头和层级，并可以与其他技术相结合以提供更好的结果。