Statistical learning theory and the Probably Approximately Correct (PAC)
criterion are the common approach to mathematical learning theory. PAC is
widely used to analyze learning problems and algorithms, and have been studied
thoroughly. Uniform worst case bounds on the convergence rate have been well
established using, e.g., VC theory or Radamacher complexity. However, in a
typical scenario the performance could be much better. In this paper, we
consider PAC learning using a somewhat different tradeoff, the error exponent -
a well established analysis method in Information Theory - which describes the
exponential behavior of the probability that the risk will exceed a certain
threshold as function of the sample size. We focus on binary classification and
find, under some stability assumptions, an improved distribution dependent
error exponent for a wide range of problems, establishing the exponential
behavior of the PAC error probability in agnostic learning. Interestingly,
under these assumptions, agnostic learning may have the same error exponent as
realizable learning. The error exponent criterion can be applied to analyze
knowledge distillation, a problem that so far lacks a theoretical analysis.

在本文中，我们通过考虑错误指数这一在信息理论中广泛应用的分析方法，研究了统计学习理论和可能近似正确（PAC）准则。在一定的稳定性假设下，我们发现二分类问题中 PAC 错误概率的指数行为，从而建立了对无知学习中可能近似正确学习的理论分析基础。有趣的是，在这些假设下，无知学习可能具有与可实现学习相同的错误指数。此错误指数准则可用于分析在缺少理论分析的知识蒸馏问题。