We uncover a surprising multilingual bias occurring in a popular class of
multimodal vision-language models (VLMs). Including an image in the query to a
LLaVA-style VLM significantly increases the likelihood of the model returning
an English response, regardless of the language of the query. This paper
investigates the causes of this loss with a two-pronged approach that combines
extensive ablation of the design space with a mechanistic analysis of the
models' internal representations of image and text inputs. Both approaches
indicate that the issue stems in the language modelling component of the LLaVA
model. Statistically, we find that switching the language backbone for a
bilingual language model has the strongest effect on reducing this error.
Mechanistically, we provide compelling evidence that visual inputs are not
mapped to a similar space as text ones, and that intervening on intermediary
attention layers can reduce this bias. Our findings provide important insights
to researchers and engineers seeking to understand the crossover between
multimodal and multilingual spaces, and contribute to the goal of developing
capable and inclusive VLMs for non-English contexts.

我们发现一种意外的多语言偏差存在于一类流行的多模态视觉语言模型（VLMs）中。将图像包含在 LLaVA 风格的 VLM 查询中，无论查询的语言如何，模型返回英文响应的可能性显着增加。本文通过对设计空间进行广泛剔除和模型对图像和文本输入进行机械分析的双重方法，调查了产生此损失的原因。两种方法都表明该问题源于 LLaVA 模型的语言建模组件。从统计上看，我们发现将语言主干切换为双语语言模型对减少此错误具有最强的效果。从机理上看，我们提供了有力的证据表明视觉输入与文本输入不被映射到类似的空间，并且对中间注意力层进行干预可以减少此偏差。我们的发现为希望了解多模态和多语言空间之间交叉的研究人员和工程师提供了重要的见解，并为开发适用于非英语环境的能力强大且包容性的 VLMs 的目标做出了贡献。