As multimodal learning finds applications in a wide variety of high-stakes
societal tasks, investigating their robustness becomes important. Existing work
has focused on understanding the robustness of vision-and-language models to
imperceptible variations on benchmark tasks. In this work, we investigate the
robustness of multimodal classifiers to cross-modal dilutions - a plausible
variation. We develop a model that, given a multimodal (image + text) input,
generates additional dilution text that (a) maintains relevance and topical
coherence with the image and existing text, and (b) when added to the original
text, leads to misclassification of the multimodal input. Via experiments on
Crisis Humanitarianism and Sentiment Detection tasks, we find that the
performance of task-specific fusion-based multimodal classifiers drops by 23.3%
and 22.5%, respectively, in the presence of dilutions generated by our model.
Metric-based comparisons with several baselines and human evaluations indicate
that our dilutions show higher relevance and topical coherence, while
simultaneously being more effective at demonstrating the brittleness of the
multimodal classifiers. Our work aims to highlight and encourage further
research on the robustness of deep multimodal models to realistic variations,
especially in human-facing societal applications. The code and other resources
are available at this https URL.

研究了多模态分类器对于跨模态稀释的抗干扰性，开发了一种在保持与图像和已有文本相关性和主题一致的前提下，能加剧多模态分类器失效的文本生成模型，并在危机人道主义和情感检测任务上进行了实验，结果表明该模型所生成的稀释文本能在更高程度上展示多模态分类器脆弱性，具有更高的相关性和主题一致性，有望进一步鼓励研究深度多模态模型对实际场景变化的抗干扰能力。