In the contemporary landscape of computer architecture, the demand for
efficient parallel programming persists, needing robust optimization
techniques. Traditional optimizing compilers have historically been pivotal in
this endeavor, adapting to the evolving complexities of modern software
systems. The emergence of Large Language Models (LLMs) raises intriguing
questions about the potential for AI-driven approaches to revolutionize code
optimization methodologies.
This paper presents a comparative analysis between two state-of-the-art Large
Language Models, GPT-4.0 and CodeLlama-70B, and traditional optimizing
compilers, assessing their respective abilities and limitations in optimizing
code for maximum efficiency. Additionally, we introduce a benchmark suite of
challenging optimization patterns and an automatic mechanism for evaluating
performance and correctness of the code generated by such tools. We used two
different prompting methodologies to assess the performance of the LLMs --
Chain of Thought (CoT) and Instruction Prompting (IP). We then compared these
results with three traditional optimizing compilers, CETUS, PLUTO and ROSE,
across a range of real-world use cases.
A key finding is that while LLMs have the potential to outperform current
optimizing compilers, they often generate incorrect code on large code sizes,
calling for automated verification methods. Our extensive evaluation across 3
different benchmarks suites shows CodeLlama-70B as the superior optimizer among
the two LLMs, capable of achieving speedups of up to 2.1x. Additionally, CETUS
is the best among the optimizing compilers, achieving a maximum speedup of
1.9x. We also found no significant difference between the two prompting
methods: Chain of Thought (Cot) and Instructing prompting (IP).

该论文通过比较分析两种先进的大型语言模型（GPT-4.0 和 CodeLlama-70B）与传统优化编译器在代码优化方面的能力和局限性，以及引入了一套挑战性的优化模式基准和用于评估生成代码性能和正确性的自动机制，发现虽然大型语言模型有超越当前优化编译器的潜力，但在大型代码上常常生成不正确的代码，需要自动化验证方法，其中 CodeLlama-70B 在两种大型语言模型中表现最佳，最大加速能达到 2.1 倍，而 CETUS 在优化编译器中表现最佳，最大加速能达到 1.9 倍。同时，两种提示方法（思考链和指令提示）没有显著差异。