Recent work has observed that pre-trained models have higher
out-of-distribution (OOD) robustness when they are exposed to less
in-distribution (ID) training data (Radford et al., 2021). In particular,
zero-shot models (e.g., GPT-3 and CLIP) have higher robustness than
conventionally fine-tuned models, but these robustness gains fade as zero-shot
models are fine-tuned on more ID data. We study this relationship between
sample efficiency and robustness -- if two models have the same ID performance,
does the model trained on fewer examples (higher sample efficiency) perform
better OOD (higher robustness)?
Surprisingly, experiments across three tasks, 23 total ID-OOD settings, and
14 models do not reveal a consistent relationship between sample efficiency and
robustness -- while models with higher sample efficiency are sometimes more
robust, most often there is no change in robustness, with some cases even
showing decreased robustness. Since results vary on a case-by-case basis, we
conduct detailed case studies of two particular ID-OOD pairs (SST-2 -> IMDb
sentiment and SNLI -> HANS) to better understand why better sample efficiency
may or may not yield higher robustness; attaining such an understanding
requires case-by-case analysis of why models are not robust on a particular
ID-OOD setting and how modeling techniques affect model capabilities.

本篇研究发现，虽然预训练模型具有更高的非分布式鲁棒性，但当零样本模型在更多的领域内数据进行精细调节时，其鲁棒性增益会逐渐减弱，因此该研究关注了不同模型的样本效率与鲁棒性之间的关系，并在案例分析中发现，获得更好的样本效率可能会带来更高的鲁棒性，也可能不会，取决于数据的情况以及建模技术的影响。

Sample-Efficient NLP 模型更加鲁棒吗？

Are Sample-Efficient NLP Models More Robust?

Contrastively trained language-image models such as CLIP, ALIGN, and BASIC
have demonstrated unprecedented robustness to multiple challenging natural
distribution shifts. Since these language-image models differ from previous
training approaches in several ways, an important question is what causes the
large robustness gains. We answer this question via a systematic experimental
investigation. Concretely, we study five different possible causes for the
robustness gains: (i) the training set size, (ii) the training distribution,
(iii) language supervision at training time, (iv) language supervision at test
time, and (v) the contrastive loss function. Our experiments show that the more
diverse training distribution is the main cause for the robustness gains, with
the other factors contributing little to no robustness. Beyond our experimental
results, we also introduce ImageNet-Captions, a version of ImageNet with
original text annotations from Flickr, to enable further controlled experiments
of language-image training.

通过实验研究，我们发现对比训练语言 - 图像模型的鲁棒性提高的主要因素是训练分布的多样性，而其他因素对鲁棒性几乎没有贡献。除了我们的实验结果，我们还介绍了 ImageNet-Captions，这是带有来自 Flickr 的原始文本注释的 ImageNet 版本，以进一步进行语言 - 图像训练的受控实验。

数据决定对比语言图像预训练（CLIP）中的分布鲁棒性

Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)

Pretrained language models (PLMs) perform poorly under adversarial attacks.
To improve the adversarial robustness, adversarial data augmentation (ADA) has
been widely adopted to cover more search space of adversarial attacks by adding
textual adversarial examples during training. However, the number of
adversarial examples for text augmentation is still extremely insufficient due
to the exponentially large attack search space. In this work, we propose a
simple and effective method to cover a much larger proportion of the attack
search space, called Adversarial and Mixup Data Augmentation (AMDA).
Specifically, AMDA linearly interpolates the representations of pairs of
training samples to form new virtual samples, which are more abundant and
diverse than the discrete text adversarial examples in conventional ADA.
Moreover, to fairly evaluate the robustness of different models, we adopt a
challenging evaluation setup, which generates a new set of adversarial examples
targeting each model. In text classification experiments of BERT and RoBERTa,
AMDA achieves significant robustness gains under two strong adversarial attacks
and alleviates the performance degradation of ADA on the clean data. Our code
is available at: this https URL .

本文提出了对抗与混合数据增强方法（AMDA）以提高预训练语言模型在对抗攻击下的鲁棒性。在 BERT 和 RoBERTa 的文本分类实验中，AMDA 在两种强对抗攻击下均取得显著的鲁棒性增益，并缓解了 ADA 在干净数据上的性能下降。