Distributed learning is commonly used for accelerating model training by
harnessing the computational capabilities of multiple-edge devices. However, in
practical applications, the communication delay emerges as a bottleneck due to
the substantial information exchange required between workers and a central
parameter server. SignSGD with majority voting (signSGD-MV) is an effective
distributed learning algorithm that can significantly reduce communication
costs by one-bit quantization. However, due to heterogeneous computational
capabilities, it fails to converge when the mini-batch sizes differ among
workers. To overcome this, we propose a novel signSGD optimizer with
\textit{federated voting} (signSGD-FV). The idea of federated voting is to
exploit learnable weights to perform weighted majority voting. The server
learns the weights assigned to the edge devices in an online fashion based on
their computational capabilities. Subsequently, these weights are employed to
decode the signs of the aggregated local gradients in such a way to minimize
the sign decoding error probability. We provide a unified convergence rate
analysis framework applicable to scenarios where the estimated weights are
known to the parameter server either perfectly or imperfectly. We demonstrate
that the proposed signSGD-FV algorithm has a theoretical convergence guarantee
even when edge devices use heterogeneous mini-batch sizes. Experimental results
show that signSGD-FV outperforms signSGD-MV, exhibiting a faster convergence
rate, especially in heterogeneous mini-batch sizes.

分布式学习中的通信延迟问题和异构的小批量大小对收敛速度的影响，通过引入权重学习及加权多数投票的 signSGD-FV 算法进行优化，该算法具有理论上的收敛保证并在异构小批量大小下表现出更快的收敛速度。

基于联合投票的 SignSGD

SignSGD with Federated Voting

Federated Learning (FL) requires frequent exchange of model parameters, which
leads to long communication delay, especially when the network environments of
clients vary greatly. Moreover, the parameter server needs to wait for the
slowest client (i.e., straggler, which may have the largest model size, lowest
computing capability or worst network condition) to upload parameters, which
may significantly degrade the communication efficiency. Commonly-used client
selection methods such as partial client selection would lead to the waste of
computing resources and weaken the generalization of the global model. To
tackle this problem, along a different line, in this paper, we advocate the
approach of model parameter dropout instead of client selection, and
accordingly propose a novel framework of Federated learning scheme with
Differential parameter Dropout (FedDD). FedDD consists of two key modules:
dropout rate allocation and uploaded parameter selection, which will optimize
the model parameter uploading ratios tailored to different clients'
heterogeneous conditions and also select the proper set of important model
parameters for uploading subject to clients' dropout rate constraints.
Specifically, the dropout rate allocation is formulated as a convex
optimization problem, taking system heterogeneity, data heterogeneity, and
model heterogeneity among clients into consideration. The uploaded parameter
selection strategy prioritizes on eliciting important parameters for uploading
to speedup convergence. Furthermore, we theoretically analyze the convergence
of the proposed FedDD scheme. Extensive performance evaluations demonstrate
that the proposed FedDD scheme can achieve outstanding performances in both
communication efficiency and model convergence, and also possesses a strong
generalization capability to data of rare classes.

基于模型参数的丢弃而非客户选择的联合学习方案，通过优化上传比例和重要参数选择，提高通信效率和模型收敛速度。