Machine learning algorithms have become ubiquitous in a number of
applications (e.g. image classification). However, due to the insufficient
measurement of traditional metrics (e.g. the coarse-grained Accuracy of each
classifier), substantial gaps are usually observed between the real-world
performance of these algorithms and their scores in standardized evaluations.
In this paper, inspired by the psychometric theories from human measurement, we
propose a task-agnostic evaluation framework Camilla, where a multi-dimensional
diagnostic metric Ability is defined for collaboratively measuring the
multifaceted strength of each machine learning algorithm. Specifically, given
the response logs from different algorithms to data samples, we leverage
cognitive diagnosis assumptions and neural networks to learn the complex
interactions among algorithms, samples and the skills (explicitly or implicitly
pre-defined) of each sample. In this way, both the abilities of each algorithm
on multiple skills and some of the sample factors (e.g. sample difficulty) can
be simultaneously quantified. We conduct extensive experiments with hundreds of
machine learning algorithms on four public datasets, and our experimental
results demonstrate that Camilla not only can capture the pros and cons of each
algorithm more precisely, but also outperforms state-of-the-art baselines on
the metric reliability, rank consistency and rank stability.

本文提出了一个基于心理测量理论的评估框架 Camilla，在多项技能上协作地度量每个机器学习算法的多方面的能力，并在公共数据集上证明了其比现有方法更具有可靠性、排名一致性和稳定性。