Knowledge distillation is a technique for improving the performance of a
simple "student" model by replacing its one-hot training labels with a
distribution over labels obtained from a complex "teacher" model. While this
simple approach has proven widely effective, a basic question remains
unresolved: why does distillation help? In this paper, we present a statistical
perspective on distillation which addresses this question, and provides a novel
connection to extreme multiclass retrieval techniques. Our core observation is
that the teacher seeks to estimate the underlying (Bayes) class-probability
function. Building on this, we establish a fundamental bias-variance tradeoff
in the student's objective: this quantifies how approximate knowledge of these
class-probabilities can significantly aid learning. Finally, we show how
distillation complements existing negative mining techniques for extreme
multiclass retrieval, and propose a unified objective which combines these
ideas.

本论文从统计角度阐述了知识蒸馏的原理，即让一个简单的 “学生” 模型依据一个复杂的 “教师” 模型得到的标签概率分布进行训练，从而提高性能，并提出了一种新颖的与极端多分类检索技术相关的方法。

蒸馏为什么有用：一个统计学的视角

Why distillation helps: a statistical perspective

Applied researchers often construct a network from a random sample of nodes
in order to infer properties of the parent network. Two of the most widely used
sampling schemes are subgraph sampling, where we sample each vertex
independently with probability $p$ and observe the subgraph induced by the
sampled vertices, and neighborhood sampling, where we additionally observe the
edges between the sampled vertices and their neighbors.
In this paper, we study the problem of estimating the number of motifs as
induced subgraphs under both models from a statistical perspective. We show
that: for any connected $h$ on $k$ vertices, to estimate $s=\mathsf{s}(h,G)$,
the number of copies of $h$ in the parent graph $G$ of maximum degree $d$, with
a multiplicative error of $\epsilon$, (a) For subgraph sampling, the optimal
sampling ratio $p$ is $\Theta_{k}(\max\{ (s\epsilon^2)^{-\frac{1}{k}}, \;
\frac{d^{k-1}}{s\epsilon^{2}} \})$, achieved by Horvitz-Thompson type of
estimators. (b) For neighborhood sampling, we propose a family of estimators,
encompassing and outperforming the Horvitz-Thompson estimator and achieving the
sampling ratio $O_{k}(\min\{ (\frac{d}{s\epsilon^2})^{\frac{1}{k-1}}, \;
\sqrt{\frac{d^{k-2}}{s\epsilon^2}}\})$. This is shown to be optimal for all
motifs with at most $4$ vertices and cliques of all sizes.
The matching minimax lower bounds are established using certain algebraic
properties of subgraph counts. These results quantify how much more informative
neighborhood sampling is than subgraph sampling, as empirically verified by
experiments on both synthetic and real-world data. We also address the issue of
adaptation to the unknown maximum degree, and study specific problems for
parent graphs with additional structures, e.g., trees or planar graphs.

本文从统计学角度研究了网络采样和子图统计中的几个问题，提出了一些估算方法，并给出了理论与实验结果。

使用图采样计算模体数量

Counting Motifs with Graph Sampling

We present a statistical perspective on boosting. Special emphasis is given
to estimating potentially complex parametric or nonparametric models, including
generalized linear and additive models as well as regression models for
survival analysis. Concepts of degrees of freedom and corresponding Akaike or
Bayesian information criteria, particularly useful for regularization and
variable selection in high-dimensional covariate spaces, are discussed as well.
The practical aspects of boosting procedures for fitting statistical models are
illustrated by means of the dedicated open-source software package mboost. This
package implements functions which can be used for model fitting, prediction
and variable selection. It is flexible, allowing for the implementation of new
boosting algorithms optimizing user-specified loss functions.

本研究从统计学的角度出发，重点讨论了利用 boosting 对复杂的参数模型或非参数模型（包括广义线性模型和加法模型等）进行有效估算的方法，以及在高维空间的正则化和变量选择中使用的信息准则，而 BOOSTING 过程的实际展示是通过开源软件包 mboost 来实现的。