We introduce Llamba, a family of efficient recurrent language models distilled from Llama-3.x into the Mamba architecture. The series includes Llamba-1B, Llamba-3B, and Llamba-8B, which achieve higher inference throughput and handle significantly larger batch sizes than Transformer-based models while maintaining comparable benchmark performance. Furthermore, Llamba demonstrates the effectiveness of cross-architecture distillation using MOHAWK (Bick et al., 2024), achieving these results with less than 0.1% of the training data typically used for models of similar size. To take full advantage of their efficiency, we provide an optimized implementation of Llamba for resource-constrained devices such as smartphones and edge platforms, offering a practical and memory-efficient alternative to Transformers. Overall, Llamba improves the tradeoff between speed, memory efficiency, and performance, making high-quality language models more accessible.

本文提出了Llamba，一种从Llama-3.x蒸馏而来的高效循环语言模型系列，解决了变压器模型在推理吞吐量和大批量处理上的局限性。Llamba通过跨架构蒸馏的方法，利用极少的训练数据实现了卓越的性能，特别针对智能手机和边缘平台进行了优化，使高质量语言模型在速度、内存效率和性能之间达成了良好平衡。

Llamba：为高效语言处理扩展的蒸馏循环模型