Zeroth-order (ZO) optimization is an emerging deep neural network (DNN) training paradigm that offers computational simplicity and memory savings. However, this seemingly promising approach faces a significant and long-ignored challenge. ZO requires generating a substantial number of Gaussian random numbers, which poses significant difficulties and even makes it infeasible for hardware platforms, such as FPGAs and ASICs. In this paper, we identify this critical issue, which arises from the mismatch between algorithm and hardware designers. To address this issue, we proposed PeZO, a perturbation-efficient ZO framework. Specifically, we design random number reuse strategies to significantly reduce the demand for random number generation and introduce a hardware-friendly adaptive scaling method to replace the costly Gaussian distribution with a uniform distribution. Our experiments show that PeZO reduces the required LUTs and FFs for random number generation by 48.6\% and 12.7\%, and saves at maximum 86\% power consumption, all without compromising training performance, making ZO optimization feasible for on-device training. To the best of our knowledge, we are the first to explore the potential of on-device ZO optimization, providing valuable insights for future research.

本研究针对零阶优化在硬件平台（如FPGA和ASIC）上面临的高高斯随机数生成需求这一挑战，提出了PeZO框架，设计了随机数重用策略并引入了适应性缩放方法，以降低硬件资源的需求和功耗。实验结果显示，PeZO显著减少了随机数生成所需的逻辑单元和功耗，使零阶优化在设备端训练成为可能，为未来研究提供了重要的见解。

硬件友好的设备端训练的扰动高效零阶优化