Soft random sampling (SRS) is a simple yet effective approach for efficient
training of large-scale deep neural networks when dealing with massive data.
SRS selects a subset uniformly at random with replacement from the full data
set in each epoch. In this paper, we conduct a theoretical and empirical
analysis of SRS. First, we analyze its sampling dynamics including data
coverage and occupancy. Next, we investigate its convergence with non-convex
objective functions and give the convergence rate. Finally, we provide its
generalization performance. We empirically evaluate SRS for image recognition
on CIFAR10 and automatic speech recognition on Librispeech and an in-house
payload dataset to demonstrate its effectiveness. Compared to existing
coreset-based data selection methods, SRS offers a better accuracy-efficiency
trade-off. Especially on real-world industrial scale data sets, it is shown to
be a powerful training strategy with significant speedup and competitive
performance with almost no additional computing cost.

软随机抽样（SRS）是一种简单而有效的方法，用于处理海量数据时高效训练大规模深度神经网络。本论文对 SRS 进行理论和实证分析，包括抽样动态、收敛性和泛化性能，以及在图像识别和自动语音识别等任务中的实证评估。相比现有的数据选择方法，SRS 在准确性和效率之间提供了更好的折衷，并在实际工业规模数据集上表现出显著加速和竞争性能，几乎不增加额外的计算成本。