Large language models (LLMs) are typically multilingual due to pretraining on
diverse multilingual corpora. But can these models relate corresponding
concepts across languages, effectively being crosslingual? This study evaluates
six state-of-the-art LLMs on inherently crosslingual tasks. We observe that
while these models show promising surface-level crosslingual abilities on
machine translation and embedding space analyses, they struggle with deeper
crosslingual knowledge transfer, revealing a crosslingual knowledge barrier in
both general (MMLU benchmark) and domain-specific (Harry Potter quiz) contexts.
We observe that simple inference-time mitigation methods offer only limited
improvement. On the other hand, we propose fine-tuning of LLMs on
mixed-language data, which effectively reduces these gaps, even when using
out-of-domain datasets like WikiText. Our findings suggest the need for
explicit optimization to unlock the full crosslingual potential of LLMs. Our
code is publicly available at
this https URL

通过评估六种最先进的大型语言模型在跨语言任务上的表现，本研究发现尽管这些模型在机器翻译和嵌入空间分析上展现了表层的跨语言能力，但在更深层次的跨语言知识转移上存在困难，揭示了跨语言知识壁垒的存在。同时提出在混合语言数据上对大型语言模型进行微调的方法，有效减少了这些差距，甚至在使用维基文本等域外数据集时也能取得良好效果。研究发现需要明确的优化方式来发挥大型语言模型的完整跨语言潜力。