Gradient clipping plays a vital role in training large-scale automatic speech
recognition (ASR) models. It is typically applied to minibatch gradients to
prevent gradient explosion, and to the individual sample gradients to mitigate
unintended memorization. This work systematically investigates the impact of a
specific granularity of gradient clipping, namely per-core clip-ping (PCC),
across training a wide range of ASR models. We empirically demonstrate that PCC
can effectively mitigate unintended memorization in ASR models. Surprisingly,
we find that PCC positively influences ASR performance metrics, leading to
improved convergence rates and reduced word error rates. To avoid tuning the
additional hyperparameter introduced by PCC, we further propose a novel
variant, adaptive per-core clipping (APCC), for streamlined optimization. Our
findings highlight the multifaceted benefits of PCC as a strategy for robust,
privacy-forward ASR model training.

渐变剪切在训练大规模自动语音识别（ASR）模型中起着至关重要的作用。本研究系统地调查了一种特定细粒度的渐变剪切方法，即每个核心剪切（PCC），对于训练各种 ASR 模型的影响。我们经验证明，PCC 可以有效地减轻 ASR 模型中的非预期记忆。令人惊讶的是，我们发现 PCC 对于 ASR 性能指标产生了积极影响，提高了收敛速度并降低了词错误率。为了避免调整 PCC 引入的额外超参数，我们进一步提出了一种新的变种，自适应每个核心剪切（APCC），用于简化优化。我们的研究结果突显了 PCC 作为一种稳健、保护隐私的 ASR 模型训练策略的多方面优势。

用 Per-core Clipping 高效训练能记忆较少且性能更好的 ASR 模型

Efficiently Train ASR Models that Memorize Less and Perform Better with  Per-core Clipping

Recent works have shown that generative sequence models (e.g., language
models) have a tendency to memorize rare or unique sequences in the training
data. Since useful models are often trained on sensitive data, to ensure the
privacy of the training data it is critical to identify and mitigate such
unintended memorization. Federated Learning (FL) has emerged as a novel
framework for large-scale distributed learning tasks. However, it differs in
many aspects from the well-studied central learning setting where all the data
is stored at the central server. In this paper, we initiate a formal study to
understand the effect of different components of canonical FL on unintended
memorization in trained models, comparing with the central learning setting.
Our results show that several differing components of FL play an important role
in reducing unintended memorization. Specifically, we observe that the
clustering of data according to users---which happens by design in FL---has a
significant effect in reducing such memorization, and using the method of
Federated Averaging for training causes a further reduction. We also show that
training with a strong user-level differential privacy guarantee results in
models that exhibit the least amount of unintended memorization.

本文探讨了联邦学习框架与中央学习环境下对非预期记忆的影响，发现在联邦学习中，根据用户对数据进行聚类的过程能够显著降低非预期记忆，使用联邦平均法进一步降低，训练具有差分隐私保证的模型则表现出最少的非预期记忆。