Transformer-based models have made remarkable advancements in various NLP areas. Nevertheless, these models often exhibit vulnerabilities when confronted with adversarial attacks. In this paper, we explore the effect of quantization on the robustness of Transformer-based models. Quantization usually involves mapping a high-precision real number to a lower-precision value, aiming at reducing the size of the model at hand. To the best of our knowledge, this work is the first application of quantization on the robustness of NLP models. In our experiments, we evaluate the impact of quantization on BERT and DistilBERT models in text classification using SST-2, Emotion, and MR datasets. We also evaluate the performance of these models against TextFooler, PWWS, and PSO adversarial attacks. Our findings show that quantization significantly improves (by an average of 18.68%) the adversarial accuracy of the models. Furthermore, we compare the effect of quantization versus that of the adversarial training approach on robustness. Our experiments indicate that quantization increases the robustness of the model by 18.80% on average compared to adversarial training without imposing any extra computational overhead during training. Therefore, our results highlight the effectiveness of quantization in improving the robustness of NLP models.

本文探讨了量化对基于Transformer模型的鲁棒性的影响，并通过对BERT和DistilBERT模型在文本分类中对SST-2、Emotion和MR数据集以及TextFooler、PWWS和PSO敌对攻击的评估，发现量化显著提高了模型的敌对准确性（提高平均18.68%）。此外，与敌对训练方法相比，量化平均提高了18.80%的模型鲁棒性而无需额外的计算开销。因此，本研究结果强调了量化在提高NLP模型鲁棒性方面的有效性。

基于转换器的文本分类器的量化对鲁棒性的影响