The large number of parameters of some prominent language models, such as
BERT, makes their fine-tuning on downstream tasks computationally intensive and
energy hungry. Previously researchers were focused on lower bit-width integer
data types for the forward propagation of language models to save memory and
computation. As for the backward propagation, however, only 16-bit
floating-point data type has been used for the fine-tuning of BERT. In this
work, we use integer arithmetic for both forward and back propagation in the
fine-tuning of BERT. We study the effects of varying the integer bit-width on
the model's metric performance. Our integer fine-tuning uses integer arithmetic
to perform forward propagation and gradient computation of linear, layer-norm,
and embedding layers of BERT. We fine-tune BERT using our integer training
method on SQuAD v1.1 and SQuAD v2., and GLUE benchmark. We demonstrate that
metric performance of fine-tuning 16-bit integer BERT matches both 16-bit and
32-bit floating-point baselines. Furthermore, using the faster and more memory
efficient 8-bit integer data type, integer fine-tuning of BERT loses an average
of 3.1 points compared to the FP32 baseline.

本研究提出使用整数算术（integer arithmetic）进行 BERT 模型的微调，通过改变整数位宽度，在 SQuAD 和 GLUE benchmark 数据集上验证了该方法的有效性，显示出 16 位整数 BERT 的指标性能与 16 位和 32 位浮点数基线相匹配。同时使用更快且更省内存的 8 位整数数据类型，整数微调 BERT 比 FP32 基线平均损失了 3.1 个指标点。

使用整数正向和反向传播微调预训练语言模型

Towards Fine-tuning Pre-trained Language Models with Integer Forward and  Backward Propagation

In this paper we present a simple and computationally efficient quantization
scheme that enables us to reduce the resolution of the parameters of a neural
network from 32-bit floating point values to 8-bit integer values. The proposed
quantization scheme leads to significant memory savings and enables the use of
optimized hardware instructions for integer arithmetic, thus significantly
reducing the cost of inference. Finally, we propose a "quantization aware"
training process that applies the proposed scheme during network training and
find that it allows us to recover most of the loss in accuracy introduced by
quantization. We validate the proposed techniques by applying them to a long
short-term memory-based acoustic model on an open-ended large vocabulary speech
recognition task.

本文提出了一种简单而高效的量化方案，将神经网络的参数分辨率从 32 位浮点值减少到 8 位整数值，从而产生了显著的内存节省和可以使用优化后的硬件指令进行整数算术操作，从而显着降低了推理成本，并通过提出的 “量化感知” 训练过程进行网络训练，发现可以恢复大部分量化引入的精度损失，并验证了该技术在一个开放式大词汇语音识别任务上的应用。

深度声学模型的高效表示与执行

On the efficient representation and execution of deep acoustic models

#SMT, or model counting for logical theories, is a well-known hard problem
that generalizes such tasks as counting the number of satisfying assignments to
a Boolean formula and computing the volume of a polytope. In the realm of
satisfiability modulo theories (SMT) there is a growing need for model counting
solvers, coming from several application domains (quantitative information
flow, static analysis of probabilistic programs). In this paper, we show a
reduction from an approximate version of #SMT to SMT.
We focus on the theories of integer arithmetic and linear real arithmetic. We
propose model counting algorithms that provide approximate solutions with
formal bounds on the approximation error. They run in polynomial time and make
a polynomial number of queries to the SMT solver for the underlying theory,
exploiting "for free" the sophisticated heuristics implemented within modern
SMT solvers. We have implemented the algorithms and used them to solve the
value problem for a model of loop-free probabilistic programs with
nondeterminism.

本篇论文介绍了一种从 SMT 到 #SMT 近似版本的约简方法，并提出了关于整数算术和线性实数算术的模型计数算法，通过对现代 SMT 求解器的高级启发式算法进行利用，运行时间为多项式级别并提供正式边界。我们使用这些算法来解决带有非确定性的无循环概率程序模型的值问题。