We address the problem of learning to benchmark the best achievable
classifier performance. In this problem the objective is to establish
statistically consistent estimates of the Bayes misclassification error rate
without having to learn a Bayes-optimal classifier. Our learning to benchmark
framework improves on previous work on learning bounds on Bayes
misclassification rate since it learns the {\it exact} Bayes error rate instead
of a bound on error rate. We propose a benchmark learner based on an ensemble
of $\epsilon$-ball estimators and Chebyshev approximation. Under a smoothness
assumption on the class densities we show that our estimator achieves an
optimal (parametric) mean squared error (MSE) rate of $O(N^{-1})$, where $N$ is
the number of samples. Experiments on both simulated and real datasets
establish that our proposed benchmark learning algorithm produces estimates of
the Bayes error that are more accurate than previous approaches for learning
bounds on Bayes error probability.

本文提出了一个学习如何对最佳分类器的表现进行基准测试的框架，实现了对贝叶斯误分类误差率的准确评估，基于一个包含估计器的集成学习器和切比雪夫逼近进行基准学习，通过实验验证，相较之前的方法精确的评估了贝叶斯误差率。

学习性能基准：从训练数据中确定最佳可达误分类率

Learning to Benchmark: Determining Best Achievable Misclassification  Error from Training Data

In the context of supervised learning, meta learning uses features, metadata
and other information to learn about the difficulty, behavior, or composition
of the problem. Using this knowledge can be useful to contextualize classifier
results or allow for targeted decisions about future data sampling. In this
paper, we are specifically interested in learning the Bayes error rate (BER)
based on a labeled data sample. Providing a tight bound on the BER that is also
feasible to estimate has been a challenge. Previous work[1] has shown that a
pairwise bound based on the sum of Henze-Penrose (HP) divergence over label
pairs can be directly estimated using a sum of Friedman-Rafsky (FR)
multivariate run test statistics. However, in situations in which the dataset
and number of classes are large, this bound is computationally infeasible to
calculate and may not be tight. Other multi-class bounds also suffer from
computationally complex estimation procedures. In this paper, we present a
generalized HP divergence measure that allows us to estimate the Bayes error
rate with log-linear computation. We prove that the proposed bound is tighter
than both the pairwise method and a bound proposed by Lin [2]. We also
empirically show that these bounds are close to the BER. We illustrate the
proposed method on the MNIST dataset, and show its utility for the evaluation
of feature reduction strategies. We further demonstrate an approach for
evaluation of deep learning architectures using the proposed bounds.

本文提出了一种广义的 HP divergence 测量方法，用于估计有标签数据样本的 Bayes error rate，并证明了该方法比基于成对测量方法和 Lin 提出的方法更紧密，展示了应用该方法进行特征还原策略和深度学习结构评估的工具。

多类别贝叶斯误差约束学习

Learning to Bound the Multi-class Bayes Error

We show that the log-likelihood of several probabilistic graphical models is
Lipschitz continuous with respect to the lp-norm of the parameters. We discuss
several implications of Lipschitz parametrization. We present an upper bound of
the Kullback-Leibler divergence that allows understanding methods that penalize
the lp-norm of differences of parameters as the minimization of that upper
bound. The expected log-likelihood is lower bounded by the negative lp-norm,
which allows understanding the generalization ability of probabilistic models.
The exponential of the negative lp-norm is involved in the lower bound of the
Bayes error rate, which shows that it is reasonable to use parameters as
features in algorithms that rely on metric spaces (e.g. classification,
dimensionality reduction, clustering). Our results do not rely on specific
algorithms for learning the structure or parameters. We show preliminary
results for activity recognition and temporal segmentation.

通过 Lipschitz parametrization，可以使用 Kullback-Leibler divergence 确定惩罚 Lipschitz 连续相关模型 lp 范数的方法；同时，负 lp-norm 的期望对数似然是其下界，并可用于理解概率模型的泛化能力和 Bayes error rate 的下界；我们还展示了活动识别和时间分割的初步结果。