Current language models have been criticised for learning language from text alone without connection between words and their meaning. Consequently, multimodal training has been proposed as a way for creating models with better language understanding by providing the lacking connection. We focus on pre-trained multimodal vision-and-language (VL) models for which there already are some results on their language understanding capabilities. An unresolved issue with evaluating the linguistic skills of these models, however, is that there is no established method for adapting them to text-only input without out-of-distribution uncertainty. To find the best approach, we investigate and compare seven possible methods for adapting three different pre-trained VL models to text-only input. Our evaluations on both GLUE and Visual Property Norms (VPN) show that care should be put into adapting VL models to zero-shot text-only tasks, while the models are less sensitive to how we adapt them to non-zero-shot tasks. We also find that the adaptation methods perform differently for different models and that unimodal model counterparts perform on par with the VL models regardless of adaptation, indicating that current VL models do not necessarily gain better language understanding from their multimodal training.

为了找到最好的方法来将视觉和语言(VL)多模态预训练模型适应于仅文本输入，我们研究并比较了适应VL模型到文本输入的七种可能的方法。我们的评估结果表明，对于零样本文本-only任务，VL模型需要小心适应，而对于非零样本任务，模型对适应方法不敏感。我们还发现，对于不同的模型，适应方法的表现各不相同，而单模态模型趋同于VL模型，提示当前的VL模型不一定从其多模态训练中获得更好的语言理解。

如何将预训练的视觉语言模型改编为仅文本输入模式？