The increasing size of large language models has posed challenges for
deployment and raised concerns about environmental impact due to high energy
consumption. In this work, we introduce BitNet, a scalable and stable 1-bit
Transformer architecture designed for large language models. Specifically, we
introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to
train 1-bit weights from scratch. Experimental results on language modeling
show that BitNet achieves competitive performance while substantially reducing
memory footprint and energy consumption, compared to state-of-the-art 8-bit
quantization methods and FP16 Transformer baselines. Furthermore, BitNet
exhibits a scaling law akin to full-precision Transformers, suggesting its
potential for effective scaling to even larger language models while
maintaining efficiency and performance benefits.

BitNet 是一种可扩展且稳定的 1 位 Transformer 架构，用于大型语言模型，通过引入 BitLinear 作为 nn.Linear 层的替代方案，BitNet 可以从零开始训练 1 位权重，实验结果表明，与最先进的 8 位量化方法和 FP16 Transformer 相比，BitNet 在语言建模方面能够达到有竞争力的性能，同时大幅减少内存占用和能源消耗，此外，BitNet 表现出与全精度 Transformer 类似的扩展规律，说明它具有在保持高效和性能优势的同时，有效扩展到更大的语言模型的潜能。