There are growing interests in adapting large-scale language models using
parameter-efficient fine-tuning methods. However, accelerating the model itself
and achieving better inference efficiency through model compression has not
been thoroughly explored yet. Model compression could provide the benefits of
reducing memory footprints, enabling low-precision computations, and ultimately
achieving cost-effective inference. To combine parameter-efficient adaptation
and model compression, we propose AlphaTuning consisting of post-training
quantization of the pre-trained language model and fine-tuning only some parts
of quantized parameters for a target task. Specifically, AlphaTuning works by
employing binary-coding quantization, which factorizes the full-precision
parameters into binary parameters and a separate set of scaling factors. During
the adaptation phase, the binary values are frozen for all tasks, while the
scaling factors are fine-tuned for the downstream task. We demonstrate that
AlphaTuning, when applied to GPT-2 and OPT, performs competitively with full
fine-tuning on a variety of downstream tasks while achieving >10x compression
ratio under 4-bit quantization and >1,000x reduction in the number of trainable
parameters.

本研究旨在将后训练量化与仅对定向任务的一些部分进行量化的精细调整相结合，以使模型更加快速且效率更高，我们提出了 AlphaTuning 控制模型的大小，通过二进制编码量化、调整二进制参数和针对下游任务进行唯一的倍数调整，证明我们的方法在 11 种下游任务上使用 4 位或更高的量化精度时，仅使用总参数数量的 1/10 进行细调。