Large-scale pre-trained multimodal transformers, such as ViLBERT and UNITER,
have propelled the state of the art in vision-and-language (V+L) research to a
new level. Although achieving impressive performance on standard tasks, to
date, it still remains unclear how robust these pre-trained models are. To
investigate, we conduct a host of thorough evaluations on existing pre-trained
models over 4 different types of V+L specific model robustness: (i) Linguistic
Variation; (ii) Logical Reasoning; (iii) Visual Content Manipulation; and (iv)
Answer Distribution Shift. Interestingly, by standard model finetuning,
pre-trained V+L models already exhibit better robustness than many
task-specific state-of-the-art methods. To further enhance model robustness, we
propose Mango, a generic and efficient approach that learns a Multimodal
Adversarial Noise GeneratOr in the embedding space to fool pre-trained V+L
models. Differing from previous studies focused on one specific type of
robustness, Mango is task-agnostic, and enables universal performance lift for
pre-trained models over diverse tasks designed to evaluate broad aspects of
robustness. Comprehensive experiments demonstrate that Mango achieves new state
of the art on 7 out of 9 robustness benchmarks, surpassing existing methods by
a significant margin. As the first comprehensive study on V+L robustness, this
work puts robustness of pre-trained models into sharper focus, pointing new
directions for future study.

通过对现有的预训练模型进行全面评估和改进，本研究提出了一种名为 Mango 的方法，在嵌入空间中学习多模态对抗性噪声生成器，使得预训练的视觉 - 语言模型的鲁棒性得到了大幅度提升，并在七项鲁棒性测试中创造了新的最高水平。