We explore the novel application of Large Language Models to code
optimization. We present a 7B-parameter transformer model trained from scratch
to optimize LLVM assembly for code size. The model takes as input unoptimized
assembly and outputs a list of compiler options to best optimize the program.
Crucially, during training, we ask the model to predict the instruction counts
before and after optimization, and the optimized code itself. These auxiliary
learning tasks significantly improve the optimization performance of the model
and improve the model's depth of understanding.
We evaluate on a large suite of test programs. Our approach achieves a 3.0%
improvement in reducing instruction counts over the compiler, outperforming two
state-of-the-art baselines that require thousands of compilations. Furthermore,
the model shows surprisingly strong code reasoning abilities, generating
compilable code 91% of the time and perfectly emulating the output of the
compiler 70% of the time.

我们探索了将大型语言模型应用于代码优化。我们提出了一个从头开始训练的 7B 参数的变换器模型，用于优化 LLVM 汇编的代码大小。该模型以未优化的汇编作为输入，并输出一系列最佳优化程序的编译器选项。在训练过程中，重要的是我们要求模型预测优化前后的指令计数和优化后的代码本身。这些辅助学习任务显著提高了模型的优化性能，并增强了模型的理解深度。我们在大量的测试程序上进行评估。我们的方法在减少指令计数方面比编译器的效果提高了 3.0%，超过了需要数千次编译的两个最先进的基线。此外，该模型显示出令人惊讶的强大代码推理能力，91% 的代码可编译，70% 的时间能完美模拟编译器的输出。