In the scenario-based evaluation of machine learning models, a key problem is
how to construct test datasets that represent various scenarios. The
methodology proposed in this paper is to construct a benchmark and attach
metadata to each test case. Then a test system can be constructed with test
morphisms that filter the test cases based on metadata to form a dataset.
The paper demonstrates this methodology with large language models for code
generation. A benchmark called ScenEval is constructed from problems in
textbooks, an online tutorial website and Stack Overflow. Filtering by scenario
is demonstrated and the test sets are used to evaluate ChatGPT for Java code
generation.
Our experiments found that the performance of ChatGPT decreases with the
complexity of the coding task. It is weakest for advanced topics like
multi-threading, data structure algorithms and recursive methods. The Java code
generated by ChatGPT tends to be much shorter than reference solution in terms
of number of lines, while it is more likely to be more complex in both
cyclomatic and cognitive complexity metrics, if the generated code is correct.
However, the generated code is more likely to be less complex than the
reference solution if the code is incorrect.

该研究论文介绍了一种基于场景的机器学习模型评估方法，并构建了一个基准测试集，用于代码生成任务的评估。实验证明，ChatGPT 在复杂的编码任务中表现最差，生成的代码行数通常比参考解决方案少，但在圈复杂度和认知复杂度方面更复杂，如果生成的代码正确，它往往比参考解决方案少复杂度，如果生成的代码不正确，则往往比参考解决方案少复杂度。

ScenEval：代码生成场景评估的基准

ScenEval: A Benchmark for Scenario-Based Evaluation of Code Generation

Predicting pedestrian behavior is one of the main challenges for intelligent
driving systems. In this paper, we present a new paradigm for evaluating
egocentric pedestrian trajectory prediction algorithms. Based on various
contextual information, we extract driving scenarios for a meaningful and
systematic approach to identifying challenges for prediction models. In this
regard, we also propose a new metric for more effective ranking within the
scenario-based evaluation. We conduct extensive empirical studies of existing
models on these scenarios to expose shortcomings and strengths of different
approaches. The scenario-based analysis highlights the importance of using
multimodal sources of information and challenges caused by inadequate modeling
of ego-motion and scale of pedestrians. To this end, we propose a novel
egocentric trajectory prediction model that benefits from multimodal sources of
data fused in an effective and efficient step-wise hierarchical fashion and two
auxiliary tasks designed to learn more robust representation of scene dynamics.
We show that our approach achieves significant improvement by up to 40% in
challenging scenarios compared to the past arts via empirical evaluation on
common benchmark datasets.

本文提出了一种基于场景进行评估的新模式，以评估自主行驶系统中行人轨迹预测算法的挑战，并通过实证评估在多模态来源数据的融合和有效的分层方式下取得了显著的改进。