Item Response Theory (IRT) has been proposed within the field of Educational Psychometrics to assess student ability as well as test question difficulty and discrimination power. More recently, IRT has been applied to evaluate machine learning algorithm performance on a single classification dataset, where the student is now an algorithm, and the test question is an observation to be classified by the algorithm. In this paper we present a modified IRT-based framework for evaluating a portfolio of algorithms across a repository of datasets, while simultaneously eliciting a richer suite of characteristics - such as algorithm consistency and anomalousness - that describe important aspects of algorithm performance. These characteristics arise from a novel inversion and reinterpretation of the traditional IRT model without requiring additional dataset feature computations. We test this framework on algorithm portfolios for a wide range of applications, demonstrating the broad applicability of this method as an insightful algorithm evaluation tool. Furthermore, the explainable nature of IRT parameters yield an increased understanding of algorithm portfolios.

在本文中，我们提出了一个基于修改过的IRT模型的框架，用于评估算法组合在数据集存储库中的性能，同时揭示算法性能的重要方面，例如一致性和异常性。我们测试了这个框架在广泛应用的算法组合上，展示了这种方法作为一种具有洞察力的算法评估工具的广泛适用性，并且IRT参数的可解释性提供了对算法组合的更深入理解。

基于项目反应理论的算法综合评估