Code generation models have increasingly become integral to aiding software
development, offering assistance in tasks such as code completion, debugging,
and code translation. Although current research has thoroughly examined the
correctness of code produced by code generation models, a vital aspect, i.e.,
the efficiency of the generated code, has often been neglected. This paper
presents EffiBench, a benchmark with 1,000 efficiency-critical coding problems
for assessing the efficiency of code generated by code generation models.
EffiBench contains a diverse set of LeetCode coding problems. Each problem is
paired with an executable human-written canonical solution. With EffiBench, we
empirically examine the capability of 21 Large Language Models (13 open-sourced
and 8 closed-sourced) in generating efficient code. The results demonstrate
that GPT-4-turbo generates the most efficient code, significantly outperforming
Palm-2-chat-bison, Claude-instant-1, Gemini-pro, GPT-4, and GPT-3.5.
Nevertheless, its code efficiency is still worse than the efficiency of
human-written canonical solutions. In particular, the average and worst
execution time of GPT-4-turbo generated code is 1.69 and 45.49 times that of
the canonical solutions.

EffiBench 是一个包含 1000 个效率关键的编码问题的基准测试，用于评估由代码生成模型生成的代码的效率，结果表明 GPT-4 Turbo 生成的代码最有效，但仍比人工编写的规范解决方案效率低。