Despite of the superb performance on a wide range of tasks, pre-trained
language models (e.g., BERT) have been proved vulnerable to adversarial texts.
In this paper, we present RoChBERT, a framework to build more Robust BERT-based
models by utilizing a more comprehensive adversarial graph to fuse Chinese
phonetic and glyph features into pre-trained representations during
fine-tuning. Inspired by curriculum learning, we further propose to augment the
training dataset with adversarial texts in combination with intermediate
samples. Extensive experiments demonstrate that RoChBERT outperforms previous
methods in significant ways: (i) robust -- RoChBERT greatly improves the model
robustness without sacrificing accuracy on benign texts. Specifically, the
defense lowers the success rates of unlimited and limited attacks by 59.43% and
39.33% respectively, while remaining accuracy of 93.30%; (ii) flexible --
RoChBERT can easily extend to various language models to solve different
downstream tasks with excellent performance; and (iii) efficient -- RoChBERT
can be directly applied to the fine-tuning stage without pre-training language
model from scratch, and the proposed data augmentation method is also low-cost.

本文介绍 RoChBERT 框架，利用包含中文音节和字形特征的更全面的对抗图，在微调过程中融合到预训练表示中，通过课程学习和对抗数据增广方法提高预训练语言模型的稳健性且不降低在正常文本上的预测准确率。RoChBERT 的性能优于以往方法，并且容易扩展到各种语言模型上，而且无需放弃预训练模型所需的时间和计算资源。

RoChBert：针对中文的鲁棒性 BERT 微调

RoChBert: Towards Robust BERT Fine-tuning for Chinese

There are two cases describing how a classifier processes input text, namely,
misclassification and correct classification. In terms of misclassified texts,
a classifier handles the texts with both incorrect predictions and adversarial
texts, which are generated to fool the classifier, which is called a victim.
Both types are misunderstood by the victim, but they can still be recognized by
other classifiers. This induces large gaps in predicted probabilities between
the victim and the other classifiers. In contrast, text correctly classified by
the victim is often successfully predicted by the others and induces small
gaps. In this paper, we propose an ensemble model based on similarity
estimation of predicted probabilities (SEPP) to exploit the large gaps in the
misclassified predictions in contrast to small gaps in the correct
classification. SEPP then corrects the incorrect predictions of the
misclassified texts. We demonstrate the resilience of SEPP in defending and
detecting adversarial texts through different types of victim classifiers,
classification tasks, and adversarial attacks.

该论文提出了一种基于相似度估计的预测概率的集成模型 (SEPP)，用于纠正分类器对误分类文本的不正确预测，并展示了该模型在不同类型的受害者分类器、分类任务和对抗攻击中识别和捍卫信息安全的鲁棒性。

SEPP: 针对对抗文本的防御和检测中的预测概率相似度估计

SEPP: Similarity Estimation of Predicted Probabilities for Defending and  Detecting Adversarial Text

NLP models are shown to suffer from robustness issues, i.e., a model's
prediction can be easily changed under small perturbations to the input. In
this work, we present a Controlled Adversarial Text Generation (CAT-Gen) model
that, given an input text, generates adversarial texts through controllable
attributes that are known to be invariant to task labels. For example, in order
to attack a model for sentiment classification over product reviews, we can use
the product categories as the controllable attribute which would not change the
sentiment of the reviews. Experiments on real-world NLP datasets demonstrate
that our method can generate more diverse and fluent adversarial texts,
compared to many existing adversarial text generation approaches. We further
use our generated adversarial examples to improve models through adversarial
training, and we demonstrate that our generated attacks are more robust against
model re-training and different model architectures.

本文提出一种名为 CAT-Gen 的 NLP 模型，通过可控属性控制生成对抗文本，并将生成的对抗例子用于提高模型的鲁棒性。实验表明，与已有的对抗文本生成方法相比，CAT-Gen 能够生成更多样化、更流畅的对抗文本，并且生成的攻击能够更好地抵御模型重新训练和不同的模型结构。