We present ALLaM: Arabic Large Language Model, a series of large language models to support the ecosystem of Arabic Language Technologies (ALT). ALLaM is carefully trained considering the values of language alignment and knowledge transfer at scale. Our autoregressive decoder-only architecture models demonstrate how second-language acquisition via vocabulary expansion and pretraining on a mixture of Arabic and English text can steer a model towards a new language (Arabic) without any catastrophic forgetting in the original language (English). Furthermore, we highlight the effectiveness of using parallel/translated data to aid the process of knowledge alignment between languages. Finally, we show that extensive alignment with human preferences can significantly enhance the performance of a language model compared to models of a larger scale with lower quality alignment. ALLaM achieves state-of-the-art performance in various Arabic benchmarks, including MMLU Arabic, ACVA, and Arabic Exams. Our aligned models improve both in Arabic and English from their base aligned models.

本研究解决了阿拉伯语言技术生态系统中阿拉伯语模型的不足，通过语言对齐和知识转移的方法训练ALLaM模型。研究表明，利用平行/翻译数据和人类偏好的广泛对齐可以显著提升语言模型的性能，使ALLaM在多项阿拉伯语基准测试中达到最先进表现。

ALLaM：阿拉伯语和英语的大型语言模型