The increasing size of large language models (LLMs) challenges their usage on
resource-constrained platforms. For example, memory on modern GPUs is
insufficient to hold LLMs that are hundreds of Gigabytes in size. Offloading is
a popular method to escape this constraint by storing weights of an LLM model
to host CPU memory and SSD, then loading each weight to GPU before every use.
In our case study of offloaded inference, we found that due to the low
bandwidth between storage devices and GPU, the latency of transferring large
model weights from its offloaded location to GPU memory becomes the critical
bottleneck with actual compute taking nearly 0% of runtime. To effectively
reduce the weight transfer latency, we propose a novel sparse format that
compresses the unstructured sparse pattern of pruned LLM weights to non-zero
values with high compression ratio and low decompression overhead. Endor
achieves this by expressing the positions of non-zero elements with a bitmap.
Compared to offloaded inference using the popular Huggingface Accelerate,
applying Endor accelerates OPT-66B by 1.70x and Llama2-70B by 1.78x. When
direct weight transfer from SSD to GPU is leveraged, Endor achieves 2.25x
speedup on OPT-66B and 2.37x speedup on Llama2-70B.

利用稀疏格式来压缩大型语言模型的权重并减少权重传输延迟，从而提高性能。