While several benefits were realized for multilingual vision-language
pretrained models, recent benchmarks across various tasks and languages showed
poor cross-lingual generalisation when multilingually pre-trained
vision-language models are applied to non-English data, with a large gap
between (supervised) English performance and (zero-shot) cross-lingual
transfer. In this work, we explore the poor performance of these models on a
zero-shot cross-lingual visual question answering (VQA) task, where models are
fine-tuned on English visual-question data and evaluated on 7 typologically
diverse languages. We improve cross-lingual transfer with three strategies: (1)
we introduce a linguistic prior objective to augment the cross-entropy loss
with a similarity-based loss to guide the model during training, (2) we learn a
task-specific subnetwork that improves cross-lingual generalisation and reduces
variance without model modification, (3) we augment training examples using
synthetic code-mixing to promote alignment of embeddings between source and
target languages. Our experiments on xGQA using the pretrained multilingual
multimodal transformers UC2 and M3P demonstrate the consistent effectiveness of
the proposed fine-tuning strategy for 7 languages, outperforming existing
transfer methods with sparse models. Code and data to reproduce our findings
are publicly available.

本研究探讨了多语言视觉语言模型在跨语言环境中表现欠佳的情况，并提出了三种策略以提高它们在零 - shot 跨语言视觉问答任务中的性能。实验证明，该 Fine-tuning 策略取得了一致的效果。