Fine-tuned transformer models have shown superior performances in many
natural language tasks. However, the large model size prohibits deploying
high-performance transformer models on resource-constrained devices. This paper
proposes a quantization-aware tensor-compressed training approach to reduce the
model size, arithmetic operations, and ultimately runtime latency of
transformer-based models. We compress the embedding and linear layers of
transformers into small low-rank tensor cores, which significantly reduces
model parameters. A quantization-aware training with learnable scale factors is
used to further obtain low-precision representations of the tensor-compressed
models. The developed approach can be used for both end-to-end training and
distillation-based training. To improve the convergence, a layer-by-layer
distillation is applied to distill a quantized and tensor-compressed student
model from a pre-trained transformer. The performance is demonstrated in two
natural language understanding tasks, showing up to $63\times$ compression
ratio, little accuracy loss and remarkable inference and training speedup.

该论文提出了一种量化感知张量压缩训练方法，通过将 Transformer 模型的嵌入层和线性层压缩成小的低秩张量核，进一步获得低精度的模型表示进行端到端和蒸馏 - based 训练，并应用层与层的蒸馏方法将预训练的 Transformer 模型转换为量化和张量压缩的学生模型以提高收敛速度，并在两个自然语言理解任务中表现出高达 63 倍的压缩比、几乎无损的精度损失和显著的推断和训练加速。