Federated averaging (FedAvg) is a widely employed paradigm for
collaboratively training models from distributed clients without sharing data.
Nowadays, the neural network has achieved remarkable success due to its
extraordinary performance, which makes it a preferred choice as the model in
FedAvg. However, the optimization problem of the neural network is often
non-convex even non-smooth. Furthermore, FedAvg always involves multiple
clients and local updates, which results in an inaccurate updating direction.
These properties bring difficulties in analyzing the convergence of FedAvg in
training neural networks. Recently, neural tangent kernel (NTK) theory has been
proposed towards understanding the convergence of first-order methods in
tackling the non-convex problem of neural networks. The deep linear neural
network is a classical model in theoretical subject due to its simple
formulation. Nevertheless, there exists no theoretical result for the
convergence of FedAvg in training the deep linear neural network. By applying
NTK theory, we make a further step to provide the first theoretical guarantee
for the global convergence of FedAvg in training deep linear neural networks.
Specifically, we prove FedAvg converges to the global minimum at a linear rate
$\mathcal{O}\big((1-\eta K /N)^t\big)$, where $t$ is the number of iterations,
$\eta$ is the learning rate, $N$ is the number of clients and $K$ is the number
of local updates. Finally, experimental evaluations on two benchmark datasets
are conducted to empirically validate the correctness of our theoretical
findings.

使用 NTK 理论进一步提供对联邦平均（FedAvg）在训练深度线性神经网络方面全局收敛的第一个理论保证，通过实验评估验证了理论发现的正确性。

深度线性神经网络联邦平均的神经切线核视角

A Neural Tangent Kernel View on Federated Averaging for Deep Linear  Neural Network

Due to the simplicity and efficiency of the first-order gradient method, it
has been widely used in training neural networks. Although the optimization
problem of the neural network is non-convex, recent research has proved that
the first-order method is capable of attaining a global minimum for training
over-parameterized neural networks, where the number of parameters is
significantly larger than that of training instances. Momentum methods,
including heavy ball method (HB) and Nesterov's accelerated method (NAG), are
the workhorse first-order gradient methods owning to their accelerated
convergence. In practice, NAG often exhibits better performance than HB.
However, current research fails to distinguish their convergence difference in
training neural networks. Motivated by this, we provide convergence analysis of
HB and NAG in training an over-parameterized two-layer neural network with ReLU
activation, through the lens of high-resolution dynamical systems and neural
tangent kernel (NTK) theory. Compared to existing works, our analysis not only
establishes tighter upper bounds of the convergence rate for both HB and NAG,
but also characterizes the effect of the gradient correction term, which leads
to the acceleration of NAG over HB. Finally, we validate our theoretical result
on three benchmark datasets.

本研究分析了在训练神经网络时，动量法中的 Heavy Ball 和 Nesterov 方法的收敛性差异，通过高分辨率动态系统和神经切向核理论对具有 ReLU 激活函数的过度参数化双层神经网络进行收敛分析。结果显示 Nesterov 方法比 Heavy Ball 方法表现更好，证明了其具有更快的收敛速度，并且通过研究梯度校正项的作用，进一步加速了 Nesterov 的收敛速度。最后，我们在三个基准数据集上验证了我们的理论结果。

超参数神经网络动量法的高分辨率动态视角

A high-resolution dynamical view on momentum methods for over-parameterized neural networks

Our goal in this paper is to exploit heteroscedastic temperature scaling as a
calibration strategy for out of distribution (OOD) detection.
Heteroscedasticity here refers to the fact that the optimal temperature
parameter for each sample can be different, as opposed to conventional
approaches that use the same value for the entire distribution. To enable this,
we propose a new training strategy called anchoring that can estimate
appropriate temperature values for each sample, leading to state-of-the-art OOD
detection performance across several benchmarks. Using NTK theory, we show that
this temperature function estimate is closely linked to the epistemic
uncertainty of the classifier, which explains its behavior. In contrast to some
of the best-performing OOD detection approaches, our method does not require
exposure to additional outlier datasets, custom calibration objectives, or
model ensembling. Through empirical studies with different OOD detection
settings -- far OOD, near OOD, and semantically coherent OOD - we establish a
highly effective OOD detection approach. Code to reproduce our results is
available at this http URL

本文目的在于利用异方差温度缩放作为一种校准策略来检测非分布数据。提出一种新的训练策略，称为锚定，可以估计每个样本的适当温度值，从而实现在几个基准测试中最先进的 OOD 检测性能。通过实验研究，我们建立了一种高度有效的 OOD 检测方法。