Distributed machine learning (DML) in mobile environments faces significant
communication bottlenecks. Gradient compression has emerged as an effective
solution to this issue, offering substantial benefits in environments with
limited bandwidth and metered data. Yet, they encounter severe performance drop
in non-IID environments due to a one-size-fits-all compression approach, which
does not account for the varying data volumes across workers. Assigning varying
compression ratios to workers with distinct data distributions and volumes is
thus a promising solution. This study introduces an analysis of distributed SGD
with non-uniform compression, which reveals that the convergence rate
(indicative of the iterations needed to achieve a certain accuracy) is
influenced by compression ratios applied to workers with differing volumes.
Accordingly, we frame relative compression ratio assignment as an $n$-variables
chi-square nonlinear optimization problem, constrained by a fixed and limited
communication budget. We propose DAGC-R, which assigns the worker handling
larger data volumes the conservative compression. Recognizing the computational
limitations of mobile devices, we DAGC-A, which are computationally less
demanding and enhances the robustness of the absolute gradient compressor in
non-IID scenarios. Our experiments confirm that both the DAGC-A and DAGC-R can
achieve better performance when dealing with highly imbalanced data volume
distribution and restricted communication.

在移动环境中，分布式机器学习面临着严重的通信瓶颈问题。梯度压缩已成为解决此问题的有效方法，在带宽有限和流量计费等环境中可以提供显著的好处。然而，在非 IID 环境中，由于一种适用于所有数据的压缩方法，它们遭遇到严重的性能下降。为具有不同数据分布和数据量的工作节点分配不同的压缩率因此是一个有前景的解决方案。本研究介绍了一种使用非均匀压缩的分布式 SGD 分析方法，揭示了收敛速率（指实现某个准确度所需的迭代次数）受应用于具有不同数据量的节点的压缩率的影响。因此，我们将相对压缩率分配作为一个 $n$ 个变量卡方非线性优化问题进行建模，并在通信开销有限的约束下进行求解。我们提出了 DAGC-R 方法，将具有较大数据量的节点分配保守的压缩率。鉴于移动设备的计算能力有限，我们还提出了 DAGC-A 方法，在非均匀数据分布的场景中具有较低的计算需求，并提高了绝对梯度压缩器的鲁棒性。实验证明，DAGC-A 和 DAGC-R 在处理高度不平衡的数据量分布和受限通信时能够获得更好的性能。

DAGC：面向移动计算的分布式机器学习数据容量感知自适应稀疏梯度压缩

DAGC: Data-Volume-Aware Adaptive Sparsification Gradient Compression for  Distributed Machine Learning in Mobile Computing

We propose using cascaded classifiers for a keyword spotting (KWS) task on
narrow-band (NB), 8kHz audio acquired in non-IID environments --- a more
challenging task than most state-of-the-art KWS systems face. We present a
model that incorporates Deep Neural Networks (DNNs), cascading,
multiple-feature representations, and multiple-instance learning. The cascaded
classifiers handle the task's class imbalance and reduce power consumption on
computationally-constrained devices via early termination. The KWS system
achieves a false negative rate of 6% at an hourly false positive rate of 0.75

本文提出了一个基于深度神经网络 (DNNs)、级联分类器、多特征表示和多实例学习的关键词检测 (KWS) 方法，用于处理在非独立同分布环境下收集的窄带语音信号。级联分类器通过早期结束的方式处理了任务中的类别不平衡问题，并可减少计算设备的能耗。实验表明，该 KWS 系统在每小时 0.75 的误报率下，误拒率达到了 6%。