Despite of the superb performance on a wide range of tasks, pre-trained
language models (e.g., BERT) have been proved vulnerable to adversarial texts.
In this paper, we present RoChBERT, a framework to build more Robust BERT-based
models by utilizing a more comprehensive adversarial graph to fuse Chinese
phonetic and glyph features into pre-trained representations during
fine-tuning. Inspired by curriculum learning, we further propose to augment the
training dataset with adversarial texts in combination with intermediate
samples. Extensive experiments demonstrate that RoChBERT outperforms previous
methods in significant ways: (i) robust -- RoChBERT greatly improves the model
robustness without sacrificing accuracy on benign texts. Specifically, the
defense lowers the success rates of unlimited and limited attacks by 59.43% and
39.33% respectively, while remaining accuracy of 93.30%; (ii) flexible --
RoChBERT can easily extend to various language models to solve different
downstream tasks with excellent performance; and (iii) efficient -- RoChBERT
can be directly applied to the fine-tuning stage without pre-training language
model from scratch, and the proposed data augmentation method is also low-cost.

本文介绍 RoChBERT 框架，利用包含中文音节和字形特征的更全面的对抗图，在微调过程中融合到预训练表示中，通过课程学习和对抗数据增广方法提高预训练语言模型的稳健性且不降低在正常文本上的预测准确率。RoChBERT 的性能优于以往方法，并且容易扩展到各种语言模型上，而且无需放弃预训练模型所需的时间和计算资源。