Deploying Large Language Models (LLMs) in streaming applications that involve
long contexts, particularly for extended dialogues and text analysis, is of
paramount importance but presents two significant challenges. Firstly, the
memory consumption is substantial during the decoding phase due to the caching
of Key and Value states (KV) of previous tokens. Secondly, attention
computation is time-consuming with a time complexity of $O(n^2)$ for the
generation of each token. In recent OpenAI DevDay (Nov 6, 2023), OpenAI
released a new model that is able to support a 128K-long document, in our
paper, we focus on the memory-efficient issue when context length $n$ is much
greater than 128K ($n \gg 2^d$). Considering a single-layer self-attention with
Query, Key, and Value matrices $Q, K, V \in \mathbb{R}^{n \times d}$, the
polynomial method approximates the attention output $T \in \mathbb{R}^{n \times
d}$. It accomplishes this by constructing $U_1, U_2 \in \mathbb{R}^{n \times
t}$ to expedite attention ${\sf Attn}(Q, K, V)$ computation within $n^{1+o(1)}$
time executions. Despite this, storing the Key and Value matrices $K, V \in
\mathbb{R}^{n \times d}$ still necessitates $O( n d)$ space, leading to
significant memory usage. In response to these challenges, we introduce a new
algorithm that only reads one pass of the data in streaming fashion. This
method employs sublinear space $o(n)$ to store three sketch matrices,
alleviating the need for exact $K, V$ storage. Notably, our algorithm exhibits
exceptional memory-efficient performance with super-long tokens. As the token
length $n$ increases, our error guarantee diminishes while the memory usage
remains nearly constant. This unique attribute underscores the potential of our
technique in efficiently handling LLMs in streaming applications.

部署大型语言模型在涉及长文本的流式应用中面临着内存消耗和时间复杂度的挑战。本文介绍了一种新的算法，通过节约存储空间并保持较低的错误率，在流式应用中高效地处理大型语言模型。

超长词注意力近似的一遍流式算法在亚线性空间中

One Pass Streaming Algorithm for Super Long Token Attention  Approximation in Sublinear Space

Deploying Large Language Models (LLMs) in streaming applications such as
multi-round dialogue, where long interactions are expected, is urgently needed
but poses two major challenges. Firstly, during the decoding stage, caching
previous tokens' Key and Value states (KV) consumes extensive memory. Secondly,
popular LLMs cannot generalize to longer texts than the training sequence
length. Window attention, where only the most recent KVs are cached, is a
natural approach -- but we show that it fails when the text length surpasses
the cache size. We observe an interesting phenomenon, namely attention sink,
that keeping the KV of initial tokens will largely recover the performance of
window attention. In this paper, we first demonstrate that the emergence of
attention sink is due to the strong attention scores towards initial tokens as
a ``sink'' even if they are not semantically important. Based on the above
analysis, we introduce StreamingLLM, an efficient framework that enables LLMs
trained with a finite length attention window to generalize to infinite
sequence lengths without any fine-tuning. We show that StreamingLLM can enable
Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language
modeling with up to 4 million tokens and more. In addition, we discover that
adding a placeholder token as a dedicated attention sink during pre-training
can further improve streaming deployment. In streaming settings, StreamingLLM
outperforms the sliding window recomputation baseline by up to 22.2x speedup.
Code and datasets are provided at this https URL

部署大型语言模型（LLMs）在流式应用中的一个研究论文，介绍了两个主要挑战和一个有效的解决方案 StreamingLLM，用于长文本的流式部署，能够在无需微调的情况下实现 LLMs 的稳定和高效的语言建模。

注意力汇聚的高效流式语言模型

Efficient Streaming Language Models with Attention Sinks

Simultaneous machine translation has recently gained traction thanks to
significant quality improvements and the advent of streaming applications.
Simultaneous translation systems need to find a trade-off between translation
quality and response time, and with this purpose multiple latency measures have
been proposed. However, latency evaluations for simultaneous translation are
estimated at the sentence level, not taking into account the sequential nature
of a streaming scenario. Indeed, these sentence-level latency measures are not
well suited for continuous stream translation resulting in figures that are not
coherent with the simultaneous translation policy of the system being assessed.
This work proposes a stream-level adaptation of the current latency measures
based on a re-segmentation approach applied to the output translation, that is
successfully evaluated on streaming conditions for a reference IWSLT task.

本文提出了基于重新划分的流级别延迟评估方法，较好地适用于连续的流翻译，并在参考 IWSLT 任务的流式条件下得到成功评估。

同时机器翻译的流级延迟评估

Stream-level Latency Evaluation for Simultaneous Machine Translation

Transformer-based acoustic modeling has achieved great suc-cess for both
hybrid and sequence-to-sequence speech recogni-tion. However, it requires
access to the full sequence, and thecomputational cost grows quadratically with
respect to the in-put sequence length. These factors limit its adoption for
stream-ing applications. In this work, we proposed a novel augmentedmemory
self-attention, which attends on a short segment of theinput sequence and a
bank of memories. The memory bankstores the embedding information for all the
processed seg-ments. On the librispeech benchmark, our proposed
methodoutperforms all the existing streamable transformer methods bya large
margin and achieved over 15% relative error reduction,compared with the widely
used LC-BLSTM baseline. Our find-ings are also confirmed on some large internal
datasets.

本文提出了一种新颖的增强记忆自注意力机制，用于 Transformer 语音识别中的流式应用，相对于现有的可流式 Transformer 方法，减小了计算量并在 Librispeech 基准测试中实现了超过 15% 的相对误差降低。