Transformer based architectures have become de-facto models used for a range
of Natural Language Processing tasks. In particular, the BERT based models
achieved significant accuracy gain for GLUE tasks, CoNLL-03 and SQuAD. However,
BERT based models have a prohibitive memory footprint and latency. As a result,
deploying BERT based models in resource constrained environments has become a
challenging task. In this work, we perform an extensive analysis of fine-tuned
BERT models using second order Hessian information, and we use our results to
propose a novel method for quantizing BERT models to ultra low precision. In
particular, we propose a new group-wise quantization scheme, and we use a
Hessian based mix-precision method to compress the model further. We
extensively test our proposed method on BERT downstream tasks of SST-2, MNLI,
CoNLL-03, and SQuAD. We can achieve comparable performance to baseline with at
most $2.3\%$ performance degradation, even with ultra-low precision
quantization down to 2 bits, corresponding up to $13\times$ compression of the
model parameters, and up to $4\times$ compression of the embedding table as
well as activations. Among all tasks, we observed the highest performance loss
for BERT fine-tuned on SQuAD. By probing into the Hessian based analysis as
well as visualization, we show that this is related to the fact that current
training/fine-tuning strategy of BERT does not converge for SQuAD.

本文通过使用 Hessian 信息对 BERT 模型进行精细调整的广泛分析，提出了一种超低精度量化 BERT 模型的新方法，并对 SST-2、MNLI、CoNLL-03 和 SQuAD 等多项下游任务进行了广泛的测试 ，证明我们的新方法实现了与基线相当的性能。