Distributed learning is commonly used for accelerating model training by
harnessing the computational capabilities of multiple-edge devices. However, in
practical applications, the communication delay emerges as a bottleneck due to
the substantial information exchange required between workers and a central
parameter server. SignSGD with majority voting (signSGD-MV) is an effective
distributed learning algorithm that can significantly reduce communication
costs by one-bit quantization. However, due to heterogeneous computational
capabilities, it fails to converge when the mini-batch sizes differ among
workers. To overcome this, we propose a novel signSGD optimizer with
\textit{federated voting} (signSGD-FV). The idea of federated voting is to
exploit learnable weights to perform weighted majority voting. The server
learns the weights assigned to the edge devices in an online fashion based on
their computational capabilities. Subsequently, these weights are employed to
decode the signs of the aggregated local gradients in such a way to minimize
the sign decoding error probability. We provide a unified convergence rate
analysis framework applicable to scenarios where the estimated weights are
known to the parameter server either perfectly or imperfectly. We demonstrate
that the proposed signSGD-FV algorithm has a theoretical convergence guarantee
even when edge devices use heterogeneous mini-batch sizes. Experimental results
show that signSGD-FV outperforms signSGD-MV, exhibiting a faster convergence
rate, especially in heterogeneous mini-batch sizes.

分布式学习中的通信延迟问题和异构的小批量大小对收敛速度的影响，通过引入权重学习及加权多数投票的 signSGD-FV 算法进行优化，该算法具有理论上的收敛保证并在异构小批量大小下表现出更快的收敛速度。

基于联合投票的 SignSGD

SignSGD with Federated Voting

A well-known issue of Batch Normalization is its significantly reduced
effectiveness in the case of small mini-batch sizes. When a mini-batch contains
few examples, the statistics upon which the normalization is defined cannot be
reliably estimated from it during a training iteration. To address this
problem, we present Cross-Iteration Batch Normalization (CBN), in which
examples from multiple recent iterations are jointly utilized to enhance
estimation quality. A challenge of computing statistics over multiple
iterations is that the network activations from different iterations are not
comparable to each other due to changes in network weights. We thus compensate
for the network weight changes via a proposed technique based on Taylor
polynomials, so that the statistics can be accurately estimated and batch
normalization can be effectively applied. On object detection and image
classification with small mini-batch sizes, CBN is found to outperform the
original batch normalization and a direct calculation of statistics over
previous iterations without the proposed compensation technique. Code is
available at this https URL .

本文提出了一种跨迭代批归一化（CBN）方法，利用多个最近的迭代中的样本共同估计统计量以增强估计质量，通过使用基于 Taylor 多项式的技术来补偿网络权重变化，以准确估计统计量，并在小型 mini-batch 情况下检测目标和图像分类任务中，CBN 显示出优于原始批归一化和对历史迭代的统计量直接计算的性能。

跨迭代批量归一化

Cross-Iteration Batch Normalization

Recent work by Brock et al. (2018) suggests that Generative Adversarial
Networks (GANs) benefit disproportionately from large mini-batch sizes.
Unfortunately, using large batches is slow and expensive on conventional
hardware. Thus, it would be nice if we could generate batches that were
effectively large though actually small. In this work, we propose a method to
do this, inspired by the use of Coreset-selection in active learning. When
training a GAN, we draw a large batch of samples from the prior and then
compress that batch using Coreset-selection. To create effectively large
batches of 'real' images, we create a cached dataset of Inception activations
of each training image, randomly project them down to a smaller dimension, and
then use Coreset-selection on those projected activations at training time. We
conduct experiments showing that this technique substantially reduces training
time and memory usage for modern GAN variants, that it reduces the fraction of
dropped modes in a synthetic dataset, and that it allows GANs to reach a new
state of the art in anomaly detection.

本文提出了一种利用 Coreset-selection 压缩训练数据，从而实现减少训练时间和内存使用的方法，并在生成对抗网络中实现了新的异常检测状态。