Pretrained foundation models offer substantial benefits for a wide range of
downstream tasks, which can be one of the most potential techniques to access
artificial general intelligence. However, scaling up foundation transformers
for maximal task-agnostic knowledge has brought about computational challenges,
especially on resource-limited devices such as mobiles. This work proposes the
first Binary Pretrained Foundation Transformer (BiPFT) for natural language
understanding (NLU) tasks, which remarkably saves 56 times operations and 28
times memory. In contrast to previous task-specific binary transformers, BiPFT
exhibits a substantial enhancement in the learning capabilities of binary
neural networks (BNNs), promoting BNNs into the era of pre-training. Benefiting
from extensive pretraining data, we further propose a data-driven binarization
method. Specifically, we first analyze the binarization error in self-attention
operations and derive the polynomials of binarization error. To simulate
full-precision self-attention, we define binarization error as binarization
residual polynomials, and then introduce low-rank estimators to model these
polynomials. Extensive experiments validate the effectiveness of BiPFTs,
surpassing task-specific baseline by 15.4% average performance on the GLUE
benchmark. BiPFT also demonstrates improved robustness to hyperparameter
changes, improved optimization efficiency, and reduced reliance on downstream
distillation, which consequently generalize on various NLU tasks and simplify
the downstream pipeline of BNNs. Our code and pretrained models are publicly
available at this https URL

首次提出了二元预训练基础模型转换器 (BiPFT) 用于自然语言理解任务，极大地节省了计算和内存资源，并通过对二值神经网络 (BNNs) 进行大规模预训练数据驱动的二值化方法，提高了学习能力，具备了 15.4% 优越的综合性能。