Practical large language model (LLM) services may involve a long system
prompt, which specifies the instructions, examples, and knowledge documents of
the task and is reused across numerous requests. However, the long system
prompt causes throughput/latency bottlenecks as the cost of generating the next
token grows w.r.t. the sequence length. This paper aims to improve the
efficiency of LLM services that involve long system prompts. Our key
observation is that handling these system prompts requires heavily redundant
memory accesses in existing causal attention computation algorithms.
Specifically, for batched requests, the cached hidden states (i.e., key-value
pairs) of system prompts are transferred from off-chip DRAM to on-chip SRAM
multiple times, each corresponding to an individual request. To eliminate such
a redundancy, we propose RelayAttention, an attention algorithm that allows
reading these hidden states from DRAM exactly once for a batch of input tokens.
RelayAttention is a free lunch: it maintains the generation quality while
requiring no model retraining, as it is based on a mathematical reformulation
of causal attention.

通过一种名为 RelayAttention 的算法，该论文提出了一种提高大型语言模型（LLM）服务效率的方法，解决了长系统提示导致的吞吐量 / 延迟瓶颈问题，该算法通过从 DRAM 准确一次性读取输入令牌批次的隐藏状态，从而消除了系统提示的冗余。

使用中继注意力实现高效的大型语言模型与长系统提示的服务

RelayAttention for Efficient Large Language Model Serving with Long  System Prompts

Many neural networks exhibit stability in their activation patterns over time
in response to inputs from sensors operating under real-world conditions. By
capitalizing on this property of natural signals, we propose a Recurrent Neural
Network (RNN) architecture called a delta network in which each neuron
transmits its value only when the change in its activation exceeds a threshold.
The execution of RNNs as delta networks is attractive because their states must
be stored and fetched at every timestep, unlike in convolutional neural
networks (CNNs). We show that a naive run-time delta network implementation
offers modest improvements on the number of memory accesses and computes, but
optimized training techniques confer higher accuracy at higher speedup. With
these optimizations, we demonstrate a 9X reduction in cost with negligible loss
of accuracy for the TIDIGITS audio digit recognition benchmark. Similarly, on
the large Wall Street Journal speech recognition benchmark even existing
networks can be greatly accelerated as delta networks, and a 5.7x improvement
with negligible loss of accuracy can be obtained through training. Finally, on
an end-to-end CNN trained for steering angle prediction in a driving dataset,
the RNN cost can be reduced by a substantial 100X.

本文提出回声神经网络（RNN）中的一种 Delta 网络，它可以利用自然信号的特性从而减少存储和访问成本，并通过优化训练技术来进一步提高准确性和速度，既可以用于 TIDIGITS 音频数字识别基准测试，也可以用于 Wall Street Journal 语音识别基准测试中，同时可以大幅度降低驾驶数据集中的方向角预测中的成本。