We consider an asynchronous decentralized learning system, which consists of
a network of connected devices trying to learn a machine learning model without
any centralized parameter server. The users in the network have their own local
training data, which is used for learning across all the nodes in the network.
The learning method consists of two processes, evolving simultaneously without
any necessary synchronization. The first process is the model update, where the
users update their local model via a fixed number of stochastic gradient
descent steps. The second process is model mixing, where the users communicate
with each other via randomized gossiping to exchange their models and average
them to reach consensus. In this work, we investigate the staleness criteria
for such a system, which is a sufficient condition for convergence of
individual user models. We show that for network scaling, i.e., when the number
of user devices $n$ is very large, if the gossip capacity of individual users
scales as $\Omega(\log n)$, we can guarantee the convergence of user models in
finite time. Furthermore, we show that the bounded staleness can only be
guaranteed by any distributed opportunistic scheme by $\Omega(n)$ scaling.

分布式异步学习系统通过模型更新和模型混合过程实现用户模型在有限时间内的收敛，并证明了当网络规模很大时，具有对数规模的用户混合能够保证用户模型的收敛，而有界的陈旧度只能通过至少与用户数量成线性关系的分布式机会方案来保证。

规模鲁棒的及时异步分散式学习

Scale-Robust Timely Asynchronous Decentralized Learning

We study the learning performance of gradient descent when the empirical risk
is weakly convex, namely, the smallest negative eigenvalue of the empirical
risk's Hessian is bounded in magnitude. By showing that this eigenvalue can
control the stability of gradient descent, generalisation error bounds are
proven that hold under a wider range of step sizes compared to previous work.
Out of sample guarantees are then achieved by decomposing the test error into
generalisation, optimisation and approximation errors, each of which can be
bounded and traded off with respect to algorithmic parameters, sample size and
magnitude of this eigenvalue. In the case of a two layer neural network, we
demonstrate that the empirical risk can satisfy a notion of local weak
convexity, specifically, the Hessian's smallest eigenvalue during training can
be controlled by the normalisation of the layers, i.e., network scaling. This
allows test error guarantees to then be achieved when the population risk
minimiser satisfies a complexity assumption. By trading off the network
complexity and scaling, insights are gained into the implicit bias of neural
network scaling, which are further supported by experimental findings.

本研究探讨了当经验风险为弱凸函数时，梯度下降的学习性能，并通过将最小负特征值应用于控制梯度下降的稳定性，从而证明了与先前的研究相比，其持有更广范围步长的一般化误差界。当经验风险满足局部弱凸性时，可以通过对网络进行归一化来控制误差，其中，两层神经网络的经验风险可以满足局部弱凸性。通过权衡网络复杂度和缩放，深入探讨了神经网络缩放的隐式偏差，并得出实验结果的支持。

使用梯度下降和弱凸损失进行学习

Learning with Gradient Descent and Weakly Convex Losses

We show that the YOLOv4 object detection neural network based on the CSP
approach, scales both up and down and is applicable to small and large networks
while maintaining optimal speed and accuracy. We propose a network scaling
approach that modifies not only the depth, width, resolution, but also
structure of the network. YOLOv4-large model achieves state-of-the-art results:
55.5% AP (73.4% AP50) for the MS COCO dataset at a speed of ~16 FPS on Tesla
V100, while with the test time augmentation, YOLOv4-large achieves 56.0% AP
(73.3 AP50). To the best of our knowledge, this is currently the highest
accuracy on the COCO dataset among any published work. The YOLOv4-tiny model
achieves 22.0% AP (42.0% AP50) at a speed of 443 FPS on RTX 2080Ti, while by
using TensorRT, batch size = 4 and FP16-precision the YOLOv4-tiny achieves 1774
FPS.

使用基于 CSP 方法的 YOLOv4 目标检测神经网络可进行极小至极大规模网络的缩放以及保持最优速度和精度，提出了一种网络缩放方法，其中 YOLOv4-large 模型在 Tesla V100 上以～16 FPS 的速度实现了 55.5% AP（73.4％AP50）的 MS COCO 数据集的最新结果，实现了 1774 FPS 的速度。通过使用 TensorRT，批量大小 = 4 和 FP16 精度，YOLOv4-tiny 实现了 22.0％AP（42.0％AP50）。