Large language models (LLMs) are useful in many NLP tasks and become more
capable with size, with the best open-source models having over 50 billion
parameters. However, using these 50B+ models requires high-end hardware, making
them inaccessible to most researchers. In this work, we investigate methods for
cost-efficient inference and fine-tuning of LLMs, comparing local and
distributed strategies. We observe that a large enough model (50B+) can run
efficiently even on geodistributed devices in a consumer-grade network. This
could allow running LLM efficiently by pooling together idle compute resources
of multiple research groups and volunteers. We address two open problems: (1)
how to perform inference and fine-tuning reliably if any device can disconnect
abruptly and (2) how to partition LLMs between devices with uneven hardware,
joining and leaving at will. In order to do that, we develop special
fault-tolerant inference algorithms and load-balancing protocols that
automatically assign devices to maximize the total system throughput. We
showcase these algorithms in Petals - a decentralized system that runs Llama 2
(70B) and BLOOM (176B) over the Internet up to 10x faster than offloading for
interactive generation. We evaluate the performance of our system in simulated
conditions and a real-world setup spanning two continents.

本研究探讨了大型语言模型在成本高效的推理和微调方面的方法，并比较了本地和分布式策略。我们开发了特殊的容错推理算法和负载平衡协议，用于自动分配设备以最大化系统总吞吐量，并展示了这些算法在 Petals 中的应用，该分散式系统能够以比离线处理快 10 倍的速度运行大型语言模型。我们通过模拟条件和跨越两大洲的真实场景对系统性能进行了评估。