The emergence of large language models (LLMs) has significantly pushed the
frontiers of program synthesis. Advancement of LLM-based program synthesis
calls for a thorough evaluation of LLM-generated code. Most evaluation
frameworks focus on the (functional) correctness of generated code; efficiency,
as an important measure of code quality, has been overlooked in existing
evaluations. In this work, we develop ENAMEL (EfficeNcy AutoMatic EvaLuator), a
rigorous and high-standard benchmark for evaluating the capability of LLMs in
generating efficient code. Firstly, we propose a new efficiency metric called
eff@k, which generalizes the pass@k metric from correctness to efficiency and
appropriately handles right-censored execution time. Furthermore, we derive an
unbiased and variance-reduced estimator of eff@k via Rao--Blackwellization; we
also provide a numerically stable implementation for the new estimator.
Secondly, to set a high-standard for efficiency evaluation, we employ a human
expert to design best algorithms and implementations as our reference solutions
of efficiency, many of which are much more efficient than existing canonical
solutions in HumanEval and HumanEval+. Moreover, to ensure a rigorous
evaluation, we employ a human expert to curate strong test case generators to
filter out wrong code and differentiate suboptimal algorithms. An extensive
study across 30 popular LLMs using our benchmark ENAMEL shows that LLMs still
fall short of generating expert-level efficient code. Using two subsets of our
problem set, we demonstrate that such deficiency is because current LLMs
struggle in designing advanced algorithms and are barely aware of
implementation optimization. Our benchmark is publicly available at
this https URL .

本研究提出了一种评估大型语言模型在生成高效代码方面能力的高标准基准 ENAMEL，通过新的效率度量标准 eff@k 和人工专家设计的参考解决方案对 30 个常用的大型语言模型进行广泛研究，发现当前的大型语言模型在设计先进算法和实现优化方面仍有不足。

LLM 生成的代码的效率如何？一个严格和高标准的基准测试

How Efficient is LLM-Generated Code? A Rigorous & High-Standard  Benchmark

Wood-leaf classification is an essential and fundamental prerequisite in the
analysis and estimation of forest attributes from terrestrial laser scanning
(TLS) point clouds,including critical measurements such as diameter at breast
height(DBH),above-ground biomass(AGB),wood volume.To address this,we introduce
the Wood-Leaf Classification Network(WLC-Net),a deep learning model derived
from PointNet++,designed to differentiate between wood and leaf points within
tree point clouds.WLC-Net enhances classification accuracy,completeness,and
speed by incorporating linearity as an inherent feature,refining the
input-output framework,and optimizing the centroid sampling technique.WLC-Net
was trained and assessed using three distinct tree species datasets,comprising
a total of 102 individual tree point clouds:21 Chinese ash trees,21 willow
trees,and 60 tropical trees.For comparative evaluation,five alternative
methods,including PointNet++,DGCNN,Krishna Moorthy's method,LeWoS, and Sun's
method,were also applied to these datasets.The classification accuracy of all
six methods was quantified using three metrics:overall accuracy(OA),mean
Intersection over Union(mIoU),and F1-score.Across all three datasets,WLC-Net
demonstrated superior performance, achieving OA scores of 0.9778, 0.9712, and
0.9508;mIoU scores of 0.9761, 0.9693,and 0.9141;and F1-scores of 0.8628,
0.7938,and 0.9019,respectively.The time costs of WLC-Net were also recorded to
evaluate the efficiency.The average processing time was 102.74s per million
points for WLC-Net.In terms of visual inspect,accuracy evaluation and
efficiency evaluation,the results suggest that WLC-Net presents a promising
approach for wood-leaf classification,distinguished by its high accuracy. In
addition,WLC-Net also exhibits strong applicability across various tree point
clouds and holds promise for further optimization.

基于深度学习的 Wood-Leaf 分类网络 (WLC-Net) 通过将线性性作为固有特征、优化输入输出结构和采样技术，提高了树木点云中木质和叶子点的分类准确性、完整性和速度。模型在三个不同的树种数据集上进行了训练和评估，并表现出优越的性能，尤其在木叶分类的准确性方面表现出色。该方法具有高应用性和进一步优化的潜力。