Recent advancements in large-scale pretrained models have significantly
improved performance across a variety of tasks in natural language processing
and computer vision. However, the extensive number of parameters in these
models necessitates substantial memory and computational resou
通过提出 Adaptive Zeroth-order Tensor-Train Adaption (AdaZeta) 框架，本文致力于改进 ZO 方法的性能和收敛性，主要关注的问题包括维度相关的 ZO 估计准确性、大规模 ZO 微调任务中的发散问题，通过详细的理论分析和实验结果论证了 AdaZeta 框架在准确性、内存效率和收敛速度方面的有效性。