Distributed learning is commonly used for accelerating model training by
harnessing the computational capabilities of multiple-edge devices. However, in
practical applications, the communication delay emerges as a bottleneck due to
the substantial information exchange required between workers and a central
parameter server. SignSGD with majority voting (signSGD-MV) is an effective
distributed learning algorithm that can significantly reduce communication
costs by one-bit quantization. However, due to heterogeneous computational
capabilities, it fails to converge when the mini-batch sizes differ among
workers. To overcome this, we propose a novel signSGD optimizer with
\textit{federated voting} (signSGD-FV). The idea of federated voting is to
exploit learnable weights to perform weighted majority voting. The server
learns the weights assigned to the edge devices in an online fashion based on
their computational capabilities. Subsequently, these weights are employed to
decode the signs of the aggregated local gradients in such a way to minimize
the sign decoding error probability. We provide a unified convergence rate
analysis framework applicable to scenarios where the estimated weights are
known to the parameter server either perfectly or imperfectly. We demonstrate
that the proposed signSGD-FV algorithm has a theoretical convergence guarantee
even when edge devices use heterogeneous mini-batch sizes. Experimental results
show that signSGD-FV outperforms signSGD-MV, exhibiting a faster convergence
rate, especially in heterogeneous mini-batch sizes.

分布式学习中的通信延迟问题和异构的小批量大小对收敛速度的影响，通过引入权重学习及加权多数投票的 signSGD-FV 算法进行优化，该算法具有理论上的收敛保证并在异构小批量大小下表现出更快的收敛速度。