Pre-trained language models (PLMs) have consistently demonstrated outstanding
performance across a diverse spectrum of natural language processing tasks.
Nevertheless, despite their success with unseen data, current PLM-based
representations often exhibit poor robustness in adversarial settings. In this
paper, we introduce RobustSentEmbed, a self-supervised sentence embedding
framework designed to improve both generalization and robustness in diverse
text representation tasks and against a diverse set of adversarial attacks.
Through the generation of high-risk adversarial perturbations and their
utilization in a novel objective function, RobustSentEmbed adeptly learns
high-quality and robust sentence embeddings. Our experiments confirm the
superiority of RobustSentEmbed over state-of-the-art representations.
Specifically, Our framework achieves a significant reduction in the success
rate of various adversarial attacks, notably reducing the BERTAttack success
rate by almost half (from 75.51\% to 38.81\%). The framework also yields
improvements of 1.59\% and 0.23\% in semantic textual similarity tasks and
various transfer tasks, respectively.

本文介绍了 RobustSentEmbed，这是一个自监督的句子嵌入框架，旨在改善自然语言处理任务中的泛化能力和对抗性环境中的鲁棒性。通过生成高风险对抗扰动并利用新颖的目标函数，RobustSentEmbed 能够熟练学习高质量且鲁棒的句子嵌入。实验证实了 RobustSentEmbed 优于最先进的表示方法的优越性，尤其是对于各种对抗性攻击，BERTAttack 的成功率从 75.51% 降低至 38.81%。该框架还在语义文本相似性任务和各种迁移任务中分别实现了 1.59% 和 0.23% 的改进。