Training modern LLMs is extremely resource intensive, and customizing them
for various deployment scenarios characterized by limited compute and memory
resources through repeated training is impractical. In this paper, we introduce
Flextron, a network architecture and post-training model optimization framework
supporting flexible model deployment. The Flextron architecture utilizes a
nested elastic structure to rapidly adapt to specific user-defined latency and
accuracy targets during inference with no additional fine-tuning required. It
is also input-adaptive, and can automatically route tokens through its
sub-networks for improved performance and efficiency. We present a
sample-efficient training method and associated routing algorithms for
systematically transforming an existing trained LLM into a Flextron model. We
evaluate Flextron on the GPT-3 and LLama-2 family of LLMs, and demonstrate
superior performance over multiple end-to-end trained variants and other
state-of-the-art elastic networks, all with a single pretraining run that
consumes a mere 7.63% tokens compared to original pretraining.

Flextron 是一种网络架构和后训练模型优化框架，用于灵活的模型部署，并通过样本高效的训练方法和路由算法将现有的 LLM 转换为 Flextron 模型，实现卓越的性能，远优于其他弹性网络和多个端到端训练变体，并且仅需要进行一轮预训练消耗原始预训练的 7.63% 令牌。