This work presents a Fully BInarized Large Language Model (FBI-LLM), demonstrating for the first time how to train a large-scale binary language model from scratch (not the partial binary or ternary LLM like BitNet b1.58) to match the performance of its full-precision counterparts (e.g., FP16 or BF16) in transformer-based LLMs. It achieves this by employing an autoregressive distillation (AD) loss with maintaining equivalent model dimensions (130M, 1.3B, 7B) and training data volume as regular LLM pretraining, while delivering competitive results in terms of perplexity and task-specific effectiveness. Intriguingly, by analyzing the training trajectory, we find that the pretrained weight is not necessary for training binarized LLMs from scratch. This research encourages a new computational framework and may facilitate the future design of specialized hardware tailored for fully 1-bit LLMs. We make all models, code, and training dataset fully accessible and transparent to support further research (Code: https://github.com/LiqunMa/FBI-LLM. Model: https://huggingface.co/LiqunMa/).

本研究提出了一种全二值化大型语言模型（FBI-LLM），首次展示如何从头开始训练大规模的二值化语言模型，使其在变压器型语言模型中达到其全精度对应模型（例如FP16或BF16）的性能。通过采用自回归蒸馏（AD）损失，在保持等效模型维度（130M，1.3B，7B）和培训数据量与常规LLM预训练相同的同时，在困惑度和特定任务的效果方面取得了有竞争力的结果。有趣的是，通过分析训练轨迹，我们发现从头开始训练二进制化LLM不需要预训练权重。这项研究鼓励了一种新的计算框架，并可能促进专门针对完全1位LLM的硬件的未来设计。我们提供了所有模型、代码和训练数据集的完全访问和透明性，以支持进一步的研究（代码：https://此处链接 模型：https://此处链接）。

FBI-LLM：通过自回归蒸馏从头开始扩展全二进制LLMs