In pursuit of more inclusive Vision-Language Models (VLMs), this study
introduces a Large Multilingual Multimodal Model called \textsc{Palo}.
\textsc{Palo} offers visual reasoning capabilities in 10 major languages,
including English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian,
Urdu, and Japanese, that span a total of $\sim$5B people (65\% of the world
population). Our approach involves a semi-automated translation approach to
adapt the multimodal instruction dataset from English to the target languages
using a fine-tuned Large Language Model, thereby ensuring high linguistic
fidelity while allowing scalability due to minimal manual effort. The
incorporation of diverse instruction sets helps us boost overall performance
across multiple languages especially those that are underrepresented like
Hindi, Arabic, Bengali, and Urdu. The resulting models are trained across three
scales (1.7B, 7B and 13B parameters) to show the generalization and scalability
where we observe substantial improvements compared to strong baselines. We also
propose the first multilingual multimodal benchmark for the forthcoming
approaches to evaluate their vision-language reasoning capabilities across
languages. Code: this https URL

本研究介绍了一个大型多语言多模态模型 Palo，它提供了 10 种主要语言（包括英语、中文、印地语、西班牙语、法语、阿拉伯语、孟加拉语、俄语、乌尔都语和日语）的视觉推理能力，总计覆盖了约 50 亿人口（世界总人口的 65%）。我们的方法采用半自动翻译方式，使用经过精调的大型语言模型将英语的多模态指令数据集转化为目标语言，以确保高语言准确性并保证可扩展性。不同语言指令的融合帮助我们提高跨多种语言的模型性能，尤其是对于印地语、阿拉伯语、孟加拉语和乌尔都语等少数被代表的语言。我们通过三个规模（17 亿、70 亿和 130 亿参数）的训练展示了该模型的普适性和可扩展性，观察到与强基线相比具有显著改进。同时，我们提出了第一个多语言多模态基准，用于评估不同语言之间的视觉推理能力。