Large-scale deep neural networks (DNNs) have achieved remarkable success in
many application scenarios. However, high computational complexity and energy
costs of modern DNNs make their deployment on edge devices challenging. Model
quantization is a common approach to deal with deployment constraints, but
searching for optimized bit-widths can be challenging. In this work, we present
Adaptive Bit-Width Quantization Aware Training (AdaQAT), a learning-based
method that automatically optimizes weight and activation signal bit-widths
during training for more efficient DNN inference. We use relaxed real-valued
bit-widths that are updated using a gradient descent rule, but are otherwise
discretized for all quantization operations. The result is a simple and
flexible QAT approach for mixed-precision uniform quantization problems.
Compared to other methods that are generally designed to be run on a pretrained
network, AdaQAT works well in both training from scratch and fine-tuning
scenarios.Initial results on the CIFAR-10 and ImageNet datasets using ResNet20
and ResNet18 models, respectively, indicate that our method is competitive with
other state-of-the-art mixed-precision quantization approaches.

本文介绍了 AdaQAT，一种基于学习的方法，用于在训练过程中自动优化深度神经网络的权重和激活信号的比特宽度，以实现更高效的推断。相比其他方法，AdaQAT 在从头开始训练和微调场景中表现良好，并在 CIFAR-10 和 ImageNet 数据集上使用 ResNet20 和 ResNet18 模型表明我们的方法与最先进的混合精度量化方法竞争力强。

AdaQAT：自适应比特宽度量化感知训练

AdaQAT: Adaptive Bit-Width Quantization-Aware Training

Soft random sampling (SRS) is a simple yet effective approach for efficient
training of large-scale deep neural networks when dealing with massive data.
SRS selects a subset uniformly at random with replacement from the full data
set in each epoch. In this paper, we conduct a theoretical and empirical
analysis of SRS. First, we analyze its sampling dynamics including data
coverage and occupancy. Next, we investigate its convergence with non-convex
objective functions and give the convergence rate. Finally, we provide its
generalization performance. We empirically evaluate SRS for image recognition
on CIFAR10 and automatic speech recognition on Librispeech and an in-house
payload dataset to demonstrate its effectiveness. Compared to existing
coreset-based data selection methods, SRS offers a better accuracy-efficiency
trade-off. Especially on real-world industrial scale data sets, it is shown to
be a powerful training strategy with significant speedup and competitive
performance with almost no additional computing cost.

软随机抽样（SRS）是一种简单而有效的方法，用于处理海量数据时高效训练大规模深度神经网络。本论文对 SRS 进行理论和实证分析，包括抽样动态、收敛性和泛化性能，以及在图像识别和自动语音识别等任务中的实证评估。相比现有的数据选择方法，SRS 在准确性和效率之间提供了更好的折衷，并在实际工业规模数据集上表现出显著加速和竞争性能，几乎不增加额外的计算成本。