LLMs have become a go-to solution not just for text generation, but also for
natural language understanding (NLU) tasks. Acquiring extensive knowledge
through language modeling on web-scale corpora, they excel on English NLU, yet
struggle to extend their NLU capabilities to underrepresented languages. In
contrast, machine translation models (MT) produce excellent multilingual
representations, resulting in strong translation performance even for
low-resource languages. MT encoders, however, lack the knowledge necessary for
comprehensive NLU that LLMs obtain through language modeling training on
immense corpora. In this work, we get the best both worlds by integrating MT
encoders directly into LLM backbones via sample-efficient self-distillation.
The resulting MT-LLMs preserve the inherent multilingual representational
alignment from the MT encoder, allowing lower-resource languages to tap into
the rich knowledge embedded in English-centric LLMs. Merging the MT encoder and
LLM in a single model, we mitigate the propagation of translation errors and
inference overhead of MT decoding inherent to discrete translation-based
cross-lingual transfer (e.g., translate-test). Evaluation spanning three
prominent NLU tasks and 127 predominantly low-resource languages renders
MT-LLMs highly effective in cross-lingual transfer. MT-LLMs substantially and
consistently outperform translate-test based on the same MT model, showing that
we truly unlock multilingual language understanding for LLMs.

通过将 MT 编码器直接集成到 LLM 主干中，我们通过样本高效的自我蒸馏获得了 MT-LLM，从而将低资源语言与以英语为中心的 LLM 中嵌入的丰富知识相结合，实现了跨语言的多语言语言理解。