We address the problem of learning to benchmark the best achievable classifier performance. In this problem the objective is to establish statistically consistent estimates of the Bayes misclassification error rate without having to learn a Bayes-optimal classifier. Our learning to benchmark framework improves on previous work on learning bounds on Bayes misclassification rate since it learns the {\it exact} Bayes error rate instead of a bound on error rate. We propose a benchmark learner based on an ensemble of $\epsilon$-ball estimators and Chebyshev approximation. Under a smoothness assumption on the class densities we show that our estimator achieves an optimal (parametric) mean squared error (MSE) rate of $O(N^{-1})$, where $N$ is the number of samples. Experiments on both simulated and real datasets establish that our proposed benchmark learning algorithm produces estimates of the Bayes error that are more accurate than previous approaches for learning bounds on Bayes error probability.

本文提出了一个学习如何对最佳分类器的表现进行基准测试的框架，实现了对贝叶斯误分类误差率的准确评估，基于一个包含估计器的集成学习器和切比雪夫逼近进行基准学习，通过实验验证，相较之前的方法精确的评估了贝叶斯误差率。

学习性能基准：从训练数据中确定最佳可达误分类率