Transformer-based long context generative models power emerging AI
applications like hour-long video understanding and project-level coding agent.
Deploying long context transformers (e.g., 100K to 10M tokens) is prohibitively
expensive compared to short context (e.g., 4K tokens) model variants. Reducing
the cost of long-context transformers is becoming a pressing research and
engineering challenge starting from the year of 2024. This work describes a
concurrent programming framework for quantitatively analyzing the efficiency
challenges in serving multiple long-context requests under limited size of GPU
high-bandwidth memory (HBM) regime. We give a detailed analysis of how all
additional computational costs, compared to 4K context, trace back to
\textit{one single source: the large size of the KV cache}. We use a 34B
GPT-3.5 level model of 50K context on A100 NVLink as a running example, and
describe how its large KV cache causes four types of deployment challenges: (1)
prefilling long inputs takes much longer compute time and GPU memory than short
inputs; (2) after prefilling, the large KV cache residing on the GPU HBM
substantially restricts the number of concurrent users being served; (3) during
decoding, repeatedly reading the KV cache from HBM to SM largely increases
latency; (4) when KV cache memory overflows, swapping it from HBM to DDR causes
significant context switching latency. We use this framework to analyze
existing works and identify possibilities of combining them to build end-to-end
systems. Overall, this work offers a foundational framework for analyzing long
context transformer deployment and identifies directions towards reducing the
inference cost of 1M context to be as cheap as 4K.

为了降低长上下文 Transformer 模型的成本并解决效率挑战，本研究提出了一种并行编程框架，用于定量分析在 GPU 高带宽内存限制下为多个长上下文请求提供服务时所面临的效率挑战，并识别出减少 1M 上下文推断成本的可能方向。