In this paper, we propose a new, simplified high probability analysis of
AdaGrad for smooth, non-convex problems. More specifically, we focus on a
particular accelerated gradient (AGD) template (Lan, 2020), through which we
recover the original AdaGrad and its variant with averaging, and prove a
convergence rate of $\mathcal O (1/ \sqrt{T})$ with high probability without
the knowledge of smoothness and variance. We use a particular version of
Freedman's concentration bound for martingale difference sequences (Kakade &
Tewari, 2008) which enables us to achieve the best-known dependence of $\log (1
/ \delta )$ on the probability margin $\delta$. We present our analysis in a
modular way and obtain a complementary $\mathcal O (1 / T)$ convergence rate in
the deterministic setting. To the best of our knowledge, this is the first high
probability result for AdaGrad with a truly adaptive scheme, i.e., completely
oblivious to the knowledge of smoothness and uniform variance bound, which
simultaneously has best-known dependence of $\log( 1/ \delta)$. We further
prove noise adaptation property of AdaGrad under additional noise assumptions.

本文提出了一种新的简化的高概率分析 AdaGrad 的方法，并证明了它在光滑非凸问题中的收敛性，并且没有光滑度和方差知识。同时，我们在附加噪声假设下进一步证明了 AdaGrad 的噪声适应性。

一类带 AdaGrad 步长的非凸算法的高概率界

High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad  Stepsize

Binary neural networks (BNNs) represent original full-precision weights and
activations into 1-bit with sign function. Since the gradient of the
conventional sign function is almost zero everywhere which cannot be used for
back-propagation, several attempts have been proposed to alleviate the
optimization difficulty by using approximate gradient. However, those
approximations corrupt the main direction of factual gradient. To this end, we
propose to estimate the gradient of sign function in the Fourier frequency
domain using the combination of sine functions for training BNNs, namely
frequency domain approximation (FDA). The proposed approach does not affect the
low-frequency information of the original sign function which occupies most of
the overall energy, and high-frequency coefficients will be ignored to avoid
the huge computational overhead. In addition, we embed a noise adaptation
module into the training phase to compensate the approximation error. The
experiments on several benchmark datasets and neural architectures illustrate
that the binary network learned using our method achieves the state-of-the-art
accuracy. Code will be available at
\textit{this https URL}.

本文提出了一种使用正弦函数组合估计傅里叶频率域中的符号函数梯度的方法，同时嵌入噪声自适应模块来弥补近似误差，结果表明该方法训练的二进制神经网络可达到最先进的精度水平。

学习频域逼近用于二值神经网络

Learning Frequency Domain Approximation for Binary Neural Networks

Crowdsourcing provides a practical way to obtain large amounts of labeled
data at a low cost. However, the annotation quality of annotators varies
considerably, which imposes new challenges in learning a high-quality model
from the crowdsourced annotations. In this work, we provide a new perspective
to decompose annotation noise into common noise and individual noise and
differentiate the source of confusion based on instance difficulty and
annotator expertise on a per-instance-annotator basis. We realize this new
crowdsourcing model by an end-to-end learning solution with two types of noise
adaptation layers: one is shared across annotators to capture their commonly
shared confusions, and the other one is pertaining to each annotator to realize
individual confusion. To recognize the source of noise in each annotation, we
use an auxiliary network to choose the two noise adaptation layers with respect
to both instances and annotators. Extensive experiments on both synthesized and
real-world benchmarks demonstrate the effectiveness of our proposed common
noise adaptation solution.

本文提出了一种新的众包模型，通过对每个实例和注释者的难度和能力进行分析，将注释噪声分解为公共噪声和个体噪声，并使用两种噪声自适应层实现端到端的学习解决方案来识别每个注释中的噪声来源。