Large language models (LLMs) predominantly employ decoder-only transformer
architectures, necessitating the retention of keys/values information for
historical tokens to provide contextual information and avoid redundant
computation. However, the substantial size and parameter volume of these LLMs
require massive GPU memory. This memory demand increases with the length of the
input text, leading to an urgent need for more efficient methods of information
storage and processing. This study introduces the Anchor-based LLM (AnLLM),
which utilizes an innovative anchor-based self-attention network (AnSAN) and
also an anchor-based inference strategy. This approach enables LLMs to compress
sequence information into an anchor token, reducing the keys/values cache and
enhancing inference efficiency. Experiments show that the AnLLM maintains
comparable accuracy with up to 99% keys/values cache reduction and up to 3.5
times faster inference. Despite a minor compromise in accuracy, the AnLLM
significantly improves computational efficiency and resource utilization,
demonstrating the potential of the anchor-based attention approach in the
context of LLMs for real-time inference in practical applications.

本研究引入了基于锚点的 LLM (AnLLM) 模型，该模型利用了一种创新的基于锚点的自注意力网络 (AnSAN) 和一种基于锚点的推理策略，将序列信息压缩到锚点令牌中，从而减少键 / 值缓存并提高推理效率。实验证明，AnLLM 在保持可比精度的同时缩减了 99% 的键 / 值缓存，并实现了高达 3.5 倍的更快推理速度。尽管在精度上有轻微折衷，AnLLM 在计算效率和资源利用方面具有显著改善，展示了锚点式注意力方法在实际应用中对于实时推理的潜力。