While the ImageNet dataset has been driving computer vision research over the
past decade, significant label noise and ambiguity have made top-1 accuracy an
insufficient measure of further progress. To address this, new label-sets and
evaluation protocols have been proposed for ImageNet showing that
state-of-the-art models already achieve over 95% accuracy and shifting the
focus on investigating why the remaining errors persist.
Recent work in this direction employed a panel of experts to manually
categorize all remaining classification errors for two selected models.
However, this process is time-consuming, prone to inconsistencies, and requires
trained experts, making it unsuitable for regular model evaluation thus
limiting its utility. To overcome these limitations, we propose the first
automated error classification framework, a valuable tool to study how modeling
choices affect error distributions. We use our framework to comprehensively
evaluate the error distribution of over 900 models. Perhaps surprisingly, we
find that across model architectures, scales, and pre-training corpora, top-1
accuracy is a strong predictor for the portion of all error types. In
particular, we observe that the portion of severe errors drops significantly
with top-1 accuracy indicating that, while it underreports a model's true
performance, it remains a valuable performance metric.
We release all our code at
this https URL .

自动化错误分类框架的综合评估表明，尽管 top-1 准确率未能完全衡量模型的真实性能，但仍然是一个有价值的性能指标，对错误类型的占比具有强大的预测能力。

ImageNet 模型错误的自动分类

Automated Classification of Model Errors on ImageNet

Finding and fixing errors is a time-consuming task not only for novice
programmers but also for expert programmers. Prior work has identified frequent
error patterns among various levels of programmers. However, the differences in
the tendencies between novices and experts have yet to be revealed. From the
knowledge of the frequent errors in each level of programmers, instructors will
be able to provide helpful advice for each level of learners. In this paper, we
propose a rule-based error classification tool to classify errors in code pairs
consisting of wrong and correct programs. We classify errors for 95,631 code
pairs and identify 3.47 errors on average, which are submitted by various
levels of programmers on an online judge system. The classified errors are used
to analyze the differences in frequent errors between novice and expert
programmers. The analyzed results show that, as for the same introductory
problems, errors made by novices are due to the lack of knowledge in
programming, and the mistakes are considered an essential part of the learning
process. On the other hand, errors made by experts are due to misunderstandings
caused by the carelessness of reading problems or the challenges of solving
problems differently than usual. The proposed tool can be used to create
error-labeled datasets and for further code-related educational research.

我们提出了一种基于规则的错误分类工具，对包含错误程序和正确程序的代码对进行错误分类，以分析初学者和专家程序员之间常见错误的差异，并为进一步的与代码相关的教育研究创建错误标记的数据集。

基于规则的错误分类方法用于分析频繁错误的差异

Rule-Based Error Classification for Analyzing Differences in Frequent  Errors

Automated feedback as students answer open-ended math questions has
significant potential in improving learning outcomes at large scale. A key part
of automated feedback systems is an error classification component, which
identifies student errors and enables appropriate, predefined feedback to be
deployed. Most existing approaches to error classification use a rule-based
method, which has limited capacity to generalize. Existing data-driven methods
avoid these limitations but specifically require mathematical expressions in
student responses to be parsed into syntax trees. This requirement is itself a
limitation, since student responses are not always syntactically valid and
cannot be converted into trees. In this work, we introduce a flexible method
for error classification using pre-trained large language models. We
demonstrate that our method can outperform existing methods in algebra error
classification, and is able to classify a larger set of student responses.
Additionally, we analyze common classification errors made by our method and
discuss limitations of automated error classification.

本研究通过使用预训练的大型语言模型，提出了一种灵活的错误分类方法，可以对更多的学生回答的数学问题进行分类，并分析了该方法的常见分类错误和自动错误分类的局限性。