We present APQ for efficient deep learning inference on resource-constrained hardware. Unlike previous methods that separately search the neural architecture, pruning policy, and quantization policy, we optimize them in a joint manner. To deal with the larger design space it brings, a promising approach is to train a quantization-aware accuracy predictor to quickly get the accuracy of the quantized model and feed it to the search engine to select the best fit. However, training this quantization-aware accuracy predictor requires collecting a large number of quantized <model, accuracy> pairs, which involves quantization-aware finetuning and thus is highly time-consuming. To tackle this challenge, we propose to transfer the knowledge from a full-precision (i.e., fp32) accuracy predictor to the quantization-aware (i.e., int8) accuracy predictor, which greatly improves the sample efficiency. Besides, collecting the dataset for the fp32 accuracy predictor only requires to evaluate neural networks without any training cost by sampling from a pretrained once-for-all network, which is highly efficient. Extensive experiments on ImageNet demonstrate the benefits of our joint optimization approach. With the same accuracy, APQ reduces the latency/energy by 2x/1.3x over MobileNetV2+HAQ. Compared to the separate optimization approach (ProxylessNAS+AMC+HAQ), APQ achieves 2.3% higher ImageNet accuracy while reducing orders of magnitude GPU hours and CO2 emission, pushing the frontier for green AI that is environmental-friendly. The code and video are publicly available.

APQ是一种用于在资源受限的硬件上高效进行深度学习推理的方法，通过联合优化神经架构、修剪策略和量化策略来处理设计空间更大的问题，同时利用全精度准确度预测器向量化至量化准确度预测器以大幅提高样本效率，其在ImageNet上的实验结果表明其可以以更低延迟/能耗降低2倍/1.3倍的前提下获得与项代号MobileNetV2+HAQ相当的准确性，并比代号为ProxylessNAS+AMC+HAQ的分别优化方法实现更高达2.3％的准确率，同时大幅减少GPU时数和CO2排放。

APQ: 网络架构、剪枝和量化策略联合搜索