There has been significant progress in recent years in the field of Natural
Language Processing thanks to the introduction of the Transformer architecture.
Current state-of-the-art models, via a large number of parameters and
pre-training on massive text corpus, have shown impressive results on several
downstream tasks. Many researchers have studied previous (non-Transformer)
models to understand their actual behavior under different scenarios, showing
that these models are taking advantage of clues or failures of datasets and
that slight perturbations on the input data can severely reduce their
performance. In contrast, recent models have not been systematically tested
with adversarial-examples in order to show their robustness under severe stress
conditions. For that reason, this work evaluates three Transformer-based models
(RoBERTa, XLNet, and BERT) in Natural Language Inference (NLI) and Question
Answering (QA) tasks to know if they are more robust or if they have the same
flaws as their predecessors. As a result, our experiments reveal that RoBERTa,
XLNet and BERT are more robust than recurrent neural network models to stress
tests for both NLI and QA tasks. Nevertheless, they are still very fragile and
demonstrate various unexpected behaviors, thus revealing that there is still
room for future improvement in this field.

本文通过对三种基于 Transformer 的模型（RoBERTa，XLNet 和 BERT）在自然语言推断（NLI）和问答（QA）任务中的评估研究，发现这些模型在脆弱性测试中表现比循环神经网络模型更鲁棒，但仍具有各种意外行为，展示了在这个领域仍有改进的空间