Transformer-based large language models (LLMs) exhibit impressive performance
in generative tasks but introduce significant challenges in real-world serving
due to inefficient use of the expensive, computation-optimized accelerators.
This mismatch arises from the autoregressive nature of LLMs, where the
generation phase comprises operators with varying resource demands.
Specifically, the attention operator is memory-intensive, exhibiting a memory
access pattern that clashes with the strengths of modern accelerators,
especially as context length increases. To enhance the efficiency and
cost-effectiveness of LLM serving, we introduce the concept of attention
offloading. This approach leverages a collection of cheap, memory-optimized
devices for the attention operator while still utilizing high-end accelerators
for other parts of the model. This heterogeneous setup ensures that each
component is tailored to its specific workload, maximizing overall performance
and cost efficiency. Our comprehensive analysis and experiments confirm the
viability of splitting the attention computation over multiple devices. Also,
the communication bandwidth required between heterogeneous devices proves to be
manageable with prevalent networking technologies. To further validate our
theory, we develop Lamina, an LLM inference system that incorporates attention
offloading. Experimental results indicate that Lamina can provide 1.48x-12.1x
higher estimated throughput per dollar than homogeneous solutions.

通过引入关注点卸载的概念，将昂贵的计算优化加速器与便宜的内存优化设备相结合，以提高大型语言模型的效率和成本效益。我们开发了 Lamina 推理系统，实验证明，相较于同质解决方案，Lamina 可以提供每美元 1.48 倍至 12.1 倍的更高预计吞吐量。