Code understanding and generation have fast become some of the most popular
applications of language models (LMs). Nonetheless, research on multilingual
aspects of Code-LMs (i.e., LMs for code generation) such as cross-lingual
transfer between different programming languages, language-specific data
augmentation, and post-hoc LM adaptation, alongside exploitation of data
sources other than the original textual content, has been much sparser than for
their natural language counterparts. In particular, most mainstream Code-LMs
have been pre-trained on source code files alone. In this work, we investigate
the prospect of leveraging readily available compiler intermediate
representations - shared across programming languages - to improve the
multilingual capabilities of Code-LMs and facilitate cross-lingual transfer.
To this end, we first compile SLTrans, a parallel dataset consisting of
nearly 4M self-contained source code files coupled with respective intermediate
representations. Next, starting from various base Code-LMs (ranging in size
from 1.1B to 7.3B parameters), we carry out continued causal language modelling
training on SLTrans, forcing the Code-LMs to (1) learn the IR language and (2)
align the IR constructs with respective constructs of various programming
languages. Our resulting models, dubbed IRCoder, display sizeable and
consistent gains across a wide variety of code generation tasks and metrics,
including prompt robustness, multilingual code completion, code understanding,
and instruction following.

通过利用编译器中间表示来改进代码语言模型的多语言能力，并促进跨语言转换，该研究探讨了在代码语言模型中利用编译器中间表示的可行性和优势，构建了一个并行数据集 SLTrans，并通过对不同规模的代码语言模型进行连续因果语言建模训练，从而在各种代码生成任务和度量标准中实现了显著和一致的提升。