Large foundation models, such as large language models, have performed
exceptionally well in various application scenarios. Building or fully
fine-tuning such large models is usually prohibitive due to either hardware
budget or lack of access to backpropagation. The zeroth-order methods offer a
promising direction for tackling this challenge, where only forward passes are
needed to update the model. This paper introduces an efficient Stochastic
Two-Point (S2P) approach within the gradient-free regime. We present the
theoretical convergence properties of S2P under the general and relaxed
smoothness assumptions. The theoretical properties also shed light on a faster
and more stable S2P variant, Accelerated S2P (AS2P), through exploiting our new
convergence properties that better represent the dynamics of deep models in
training. Our comprehensive empirical results show that AS2P is highly
effective in optimizing objectives for large deep models, including language
models, and outperforms standard methods across various model types and scales,
with 2 $\times$ speed-up in training over most conducted tasks.

该研究介绍了零阶方法在大型深度模型优化中的应用，提出了一种高效的随机双点（S2P）方法及其加速变种（AS2P），通过利用新的收敛性质，成功优化了大型深层模型的目标函数，相对于标准方法在训练中加快了 2 倍的速度。

深层模型零阶优化的随机两点方法

Stochastic Two Points Method for Deep Model Zeroth-order Optimization

The widespread practice of fine-tuning pretrained large language models
(LLMs) on domain-specific data faces two major challenges in memory and
privacy. First, as the size of LLMs continue to grow, encompassing billions of
parameters, the memory demands of gradient-based training methods via
backpropagation become prohibitively high. Second, given the tendency of LLMs
to memorize and disclose sensitive training data, the privacy of fine-tuning
data must be respected. To this end, we explore the potential of zeroth-order
methods in differentially private optimization for fine-tuning LLMs.
Zeroth-order methods, which rely solely on forward passes, substantially reduce
memory consumption during training. However, directly combining them with
standard differential privacy mechanism poses dimension-dependent complexity.
To bridge the gap, we introduce DPZero, a novel differentially private
zeroth-order algorithm with nearly dimension-independent rates. Our theoretical
analysis reveals that its complexity hinges primarily on the problem's
intrinsic dimension and exhibits only a logarithmic dependence on the ambient
dimension. This renders DPZero a highly practical option for real-world LLMs
deployments.

通过零阶方法进行微调预训练语言模型，以解决精调大型语言模型在内存和隐私方面面临的挑战。DPZero 是一个新的具有几乎维度无关率的差分隐私零阶算法，对于实际中的语言模型部署具有高度实用性。

DPZero: 无维度限制和差分隐私的零阶优化

DPZero: Dimension-Independent and Differentially Private Zeroth-Order  Optimization

Alternating direction method of multipliers (ADMM) is a popular optimization
tool for the composite and constrained problems in machine learning. However,
in many machine learning problems such as black-box attacks and bandit
feedback, ADMM could fail because the explicit gradients of these problems are
difficult or infeasible to obtain. Zeroth-order (gradient-free) methods can
effectively solve these problems due to that the objective function values are
only required in the optimization. Recently, though there exist a few
zeroth-order ADMM methods, they build on the convexity of objective function.
Clearly, these existing zeroth-order methods are limited in many applications.
In the paper, thus, we propose a class of fast zeroth-order stochastic ADMM
methods (i.e., ZO-SVRG-ADMM and ZO-SAGA-ADMM) for solving nonconvex problems
with multiple nonsmooth penalties, based on the coordinate smoothing gradient
estimator. Moreover, we prove that both the ZO-SVRG-ADMM and ZO-SAGA-ADMM have
convergence rate of $O(1/T)$, where $T$ denotes the number of iterations. In
particular, our methods not only reach the best convergence rate $O(1/T)$ for
the nonconvex optimization, but also are able to effectively solve many complex
machine learning problems with multiple regularized penalties and constraints.
Finally, we conduct the experiments of black-box binary classification and
structured adversarial attack on black-box deep neural network to validate the
efficiency of our algorithms.

本文提出了一种基于坐标平滑梯度估计器的快速零阶随机 ADMM 方法（即 ZO-SVRG-ADMM 和 ZO-SAGA-ADMM），用于解决具有多个非光滑惩罚的非凸问题，证明了这两种方法的收敛速率为 $O（1 / T）$，可以有效地解决许多复杂的机器学习问题。