In the era of widespread public use of AI systems across various domains,
ensuring adversarial robustness has become increasingly vital to maintain
safety and prevent undesirable errors. Researchers have curated various
adversarial datasets (through perturbations) for capturing model deficiencies
that cannot be revealed in standard benchmark datasets. However, little is
known about how these adversarial examples differ from the original data
points, and there is still no methodology to measure the intended and
unintended consequences of those adversarial transformations. In this research,
we conducted a systematic survey of existing quantifiable metrics that describe
text instances in NLP tasks, among dimensions of difficulty, diversity, and
disagreement. We selected several current adversarial effect datasets and
compared the distributions between the original and their adversarial
counterparts. The results provide valuable insights into what makes these
datasets more challenging from a metrics perspective and whether they align
with underlying assumptions.

在广泛应用于各个领域的 AI 系统的时代，确保对抗性鲁棒性变得越来越重要，以维护安全性并防止不可取的错误。本研究对描述 NLP 任务中文本实例的现有可量化指标进行了系统调查，并选择了几个当前的对抗效应数据集，比较了原始数据与对抗样本之间的分布差异，研究结果揭示了这些数据集在度量角度上更具挑战性的原因及其与基本假设的一致性。