Incremental learning is one paradigm to enable model building and updating at
scale with streaming data. For end-to-end automatic speech recognition (ASR)
tasks, the absence of human annotated labels along with the need for privacy
preserving policies for model building makes it a daunting challenge. Motivated
by these challenges, in this paper we use a cloud based framework for
production systems to demonstrate insights from privacy preserving incremental
learning for automatic speech recognition (ILASR). By privacy preserving, we
mean, usage of ephemeral data which are not human annotated. This system is a
step forward for production levelASR models for incremental/continual learning
that offers near real-time test-bed for experimentation in the cloud for
end-to-end ASR, while adhering to privacy-preserving policies. We show that the
proposed system can improve the production models significantly(3%) over a new
time period of six months even in the absence of human annotated labels with
varying levels of weak supervision and large batch sizes in incremental
learning. This improvement is 20% over test sets with new words and phrases in
the new time period. We demonstrate the effectiveness of model building in a
privacy-preserving incremental fashion for ASR while further exploring the
utility of having an effective teacher model and use of large batch sizes.

本文提出了一种用于自动语音识别的隐私保护增量学习框架，使用了未经人类标注的临时数据和大批量训练，显示出该框架在云端环境中的高实时性和实际应用性。该框架提高了生产模型的表现，尤其是在新时间段的新词和短语测试中。

ILASR: 面向生产规模的隐私保护增量学习对于自动语音识别

ILASR: Privacy-Preserving Incremental Learning for Automatic Speech Recognition at Production Scale

State-of-the-art optimization is steadily shifting towards massively parallel
pipelines with extremely large batch sizes. As a consequence, CPU-bound
preprocessing and disk/memory/network operations have emerged as new
performance bottlenecks, as opposed to hardware-accelerated gradient
computations. In this regime, a recently proposed approach is data echoing
(Choi et al., 2019), which takes repeated gradient steps on the same batch
while waiting for fresh data to arrive from upstream. We provide the first
convergence analyses of "data-echoed" extensions of common optimization
methods, showing that they exhibit provable improvements over their synchronous
counterparts. Specifically, we show that in convex optimization with stochastic
minibatches, data echoing affords speedups on the curvature-dominated part of
the convergence rate, while maintaining the optimal statistical rate.

该研究文章提出了一种名为 “数据回响” 的新优化方法，通过在等待新数据到达的同时在同一批次上重复梯度步骤来克服大规模并行管道中的性能瓶颈。而且，该方法在凸优化中的随机小批量优化以及在其他正在被大规模并行化的领域中得到了证明，展现出明显的加速和优越性能。

滞后数据管道的随机优化

Stochastic Optimization with Laggard Data Pipelines

We provide an improved analysis of normalized SGD showing that adding
momentum provably removes the need for large batch sizes on non-convex
objectives. Then, we consider the case of objectives with bounded second
derivative and show that in this case a small tweak to the momentum formula
allows normalized SGD with momentum to find an $\epsilon$-critical point in
$O(1/\epsilon^{3.5})$ iterations, matching the best-known rates without
accruing any logarithmic factors or dependence on dimension. We also provide an
adaptive method that automatically improves convergence rates when the variance
in the gradients is small. Finally, we show that our method is effective when
employed on popular large scale tasks such as ResNet-50 and BERT pretraining,
matching the performance of the disparate methods used to get state-of-the-art
results on both tasks.

本研究提出使用 momentum 相对于 normalized SGD 来解决 non-convex issues，若 objectives 带有有限的第二导数，采用带的动量公式提高了收敛速度，且适用于大规模任务如 ResNet-50 和 BERT pretraining。