Random label noises (or observational noises) widely exist in practical
machine learning settings. While previous studies primarily focus on the
affects of label noises to the performance of learning, our work intends to
investigate the implicit regularization effects of the label noises, under
mini-batch sampling settings of stochastic gradient descent (SGD), with
assumptions that label noises are unbiased. Specifically, we analyze the
learning dynamics of SGD over the quadratic loss with unbiased label noises,
where we model the dynamics of SGD as a stochastic differentiable equation
(SDE) with two diffusion terms (namely a Doubly Stochastic Model). While the
first diffusion term is caused by mini-batch sampling over the
(label-noiseless) loss gradients as many other works on SGD, our model
investigates the second noise term of SGD dynamics, which is caused by
mini-batch sampling over the label noises, as an implicit regularizer. Our
theoretical analysis finds such implicit regularizer would favor some
convergence points that could stabilize model outputs against perturbation of
parameters (namely inference stability). Though similar phenomenon have been
investigated, our work doesn't assume SGD as an Ornstein-Uhlenbeck like process
and achieve a more generalizable result with convergence of approximation
proved. To validate our analysis, we design two sets of empirical studies to
analyze the implicit regularizer of SGD with unbiased random label noises for
deep neural networks training and linear regression.

研究了标签噪声（或观测噪声）对于随机梯度下降（SGD）的小型抽样设置具有隐式正则化效果的影响，并对其进行了理论分析与实验验证。

双重随机模型：无偏标签噪声的学习和推断稳定性

Doubly Stochastic Models: Learning with Unbiased Label Noises and  Inference Stability

Self-paced learning (SPL) mimics the cognitive mechanism of humans and
animals that gradually learns from easy to hard samples. One key issue in SPL
is to obtain better weighting strategy that is determined by minimizer
function. Existing methods usually pursue this by artificially designing the
explicit form of SPL regularizer. In this paper, we focus on the minimizer
function, and study a group of new regularizer, named self-paced implicit
regularizer that is deduced from robust loss function. Based on the convex
conjugacy theory, the minimizer function for self-paced implicit regularizer
can be directly learned from the latent loss function, while the analytic form
of the regularizer can be even known. A general framework (named SPL-IR) for
SPL is developed accordingly. We demonstrate that the learning procedure of
SPL-IR is associated with latent robust loss functions, thus can provide some
theoretical inspirations for its working mechanism. We further analyze the
relation between SPL-IR and half-quadratic optimization. Finally, we implement
SPL-IR to both supervised and unsupervised tasks, and experimental results
corroborate our ideas and demonstrate the correctness and effectiveness of
implicit regularizers.

本论文研究了一个新的自适应学习方法 —— 自适应隐式规则化器，并提出了用于该方法的通用框架。该框架使用基于凸共轭理论的隐式损失函数学习最小化函数，从而能够更好地权衡避免过拟合和学习难度逐渐加大之间的关系。同时，研究还发现该方法与半二次优化方法的关系，并通过实验验证了其正确性和有效性。