The robustness of multimodal deep learning models to realistic changes in the
input text is critical for their applicability to important tasks such as
text-to-image retrieval and cross-modal entailment. To measure robustness,
several existing approaches edit the text data, but do so without leveraging
the cross-modal information present in multimodal data. Information from the
visual modality, such as color, size, and shape, provide additional attributes
that users can include in their inputs. Thus, we propose cross-modal attribute
insertions as a realistic perturbation strategy for vision-and-language data
that inserts visual attributes of the objects in the image into the
corresponding text (e.g., "girl on a chair" to "little girl on a wooden
chair"). Our proposed approach for cross-modal attribute insertions is modular,
controllable, and task-agnostic. We find that augmenting input text using
cross-modal insertions causes state-of-the-art approaches for text-to-image
retrieval and cross-modal entailment to perform poorly, resulting in relative
drops of 15% in MRR and 20% in $F_1$ score, respectively. Crowd-sourced
annotations demonstrate that cross-modal insertions lead to higher quality
augmentations for multimodal data than augmentations using text-only data, and
are equivalent in quality to original examples. We release the code to
encourage robustness evaluations of deep vision-and-language models:
this https URL

本文提出了一种跨模态属性插入的策略，该策略将图像中的视觉属性插入到文本数据中，用于深度视觉和语言模型的鲁棒性评估，并发现相对于纯文本数据，跨模态属性插入提高了深度视觉和语言模型的数据增广质量。