The rapid scaling of language models is motivating research using
low-bitwidth quantization. In this work, we propose a novel binarization
technique for Transformers applied to machine translation (BMT), the first of
its kind. We identify and address the problem of inflated dot-product variance
when using one-bit weights and activations. Specifically, BMT leverages
additional LayerNorms and residual connections to improve binarization quality.
Experiments on the WMT dataset show that a one-bit weight-only Transformer can
achieve the same quality as a float one, while being 16x smaller in size.
One-bit activations incur varying degrees of quality drop, but mitigated by the
proposed architectural changes. We further conduct a scaling law study using
production-scale translation datasets, which shows that one-bit weight
Transformers scale and generalize well in both in-domain and out-of-domain
settings. Implementation in JAX/Flax will be open sourced.

本文提出了一种新的二值化技术，基于一位权重和激活函数的 Transformer 应用于机器翻译，实验证明其可以达到与浮点型 Transformer 相同的质量，同时大小只有其 16 倍，并通过额外的 LayerNorms 和残差连接改进了二值化的质量。此外，作者在生产规模的翻译数据集上进行了一项标尺研究，表明一位权重 Transformer 在域内和域外环境下均具有可扩展性和良好的泛化效果。