The rapid development of large pre-trained language models has greatly
increased the demand for model compression techniques, among which quantization
is a popular solution. In this paper, we propose BinaryBERT, which pushes BERT
quantization to the limit by weight binarization. We find that a binary BERT is
hard to be trained directly than a ternary counterpart due to its complex and
irregular loss landscape. Therefore, we propose ternary weight splitting, which
initializes BinaryBERT by equivalently splitting from a half-sized ternary
network. The binary model thus inherits the good performance of the ternary
one, and can be further enhanced by fine-tuning the new architecture after
splitting. Empirical results show that our BinaryBERT has only a slight
performance drop compared with the full-precision model while being 24x
smaller, achieving the state-of-the-art compression results on the GLUE and
SQuAD benchmarks.

本文提出了 BinaryBERT，一种采用权重二值化的方法将 BERT 压缩到极限。论文发现，二进制 BERT 比三进制的更难以直接进行训练，因此提出了三进制权重分裂来初始化 BinaryBERT 网络。实验结果表明，BinaryBERT 模型只有轻微的性能损失，同时体积缩小了 24 倍，达到了 GLUE 和 SQuAD 基准测试的最先进压缩结果。