The increasing size of generative Pre-trained Language Models (PLMs) has
greatly increased the demand for model compression. Despite various methods to
compress BERT or its variants, there are few attempts to compress generative
PLMs, and the underlying difficulty remains unclear. In this paper, we compress
generative PLMs by quantization. We find that previous quantization methods
fail on generative tasks due to the \textit{homogeneous word embeddings} caused
by reduced capacity, and \textit{varied distribution of weights}.
Correspondingly, we propose a token-level contrastive distillation to learn
distinguishable word embeddings, and a module-wise dynamic scaling to make
quantizers adaptive to different modules. Empirical results on various tasks
show that our proposed method outperforms the state-of-the-art compression
methods on generative PLMs by a clear margin. With comparable performance with
the full-precision models, we achieve 14.4x and 13.4x compression rates on
GPT-2 and BART, respectively.

本文介绍了一种基于量化的压缩生成式语言模型的方法，该方法使用了基于 token 的对比 distillation 技术学习可区分的单词嵌入，并提出了一个基于模块的动态缩放技术，适应不同模块的量化器，实验结果表明该方法在各种任务上性能优于基准方法，并且实现了 14.4x 和 13.4x 的压缩率，与全精度模型具有可比性。

通过量化实现生成预训练语言模型的压缩

Compression of Generative Pre-trained Language Models via Quantization

Deep neural network-based image classification can be misled by adversarial
examples with small and quasi-imperceptible perturbations. Furthermore, the
adversarial examples created on one classification model can also fool another
different model. The transferability of the adversarial examples has recently
attracted a growing interest since it makes black-box attacks on classification
models feasible. As an extension of classification, semantic segmentation has
also received much attention towards its adversarial robustness. However, the
transferability of adversarial examples on segmentation models has not been
systematically studied. In this work, we intensively study this topic. First,
we explore the overfitting phenomenon of adversarial examples on classification
and segmentation models. In contrast to the observation made on classification
models that the transferability is limited by overfitting to the source model,
we find that the adversarial examples on segmentations do not always overfit
the source models. Even when no overfitting is presented, the transferability
of adversarial examples is limited. We attribute the limitation to the
architectural traits of segmentation models, i.e., multi-scale object
recognition. Then, we propose a simple and effective method, dubbed dynamic
scaling, to overcome the limitation. The high transferability achieved by our
method shows that, in contrast to the observations in previous work,
adversarial examples on a segmentation model can be easy to transfer to other
segmentation models. Our analysis and proposals are supported by extensive
experiments.

该研究探讨了深度神经网络中对抗样本的传递性问题，研究发现在语义分割模型中，由于其具有多尺度目标识别的特性，传递性存在限制。提出了一种称之为动态尺度调节的简单而有效的方法来解决该问题。经过大量实验证明，该方法能够实现高效的传递性。