How much can you say about the gradient of a neural network without computing
a loss or knowing the label? This may sound like a strange question: surely the
answer is "very little." However, in this paper, we show that gradients are
more structured than previously thought. Gradients lie in a predictable
low-dimensional subspace which depends on the network architecture and incoming
features. Exploiting this structure can significantly improve gradient-free
optimization schemes based on directional derivatives, which have struggled to
scale beyond small networks trained on toy datasets. We study how to narrow the
gap in optimization performance between methods that calculate exact gradients
and those that use directional derivatives. Furthermore, we highlight new
challenges in overcoming the large gap between optimizing with exact gradients
and guessing the gradients.

神经网络的梯度具有比以前认为的更多结构，研究中探讨了梯度在可预测的低维子空间中的特点以及如何利用这种结构改进基于方向导数的无梯度优化方案。同时，突出了在最大程度地减小准确梯度计算方法和猜测梯度方法之间的优化性能差距方面所面临的新挑战。

如何猜测梯度

How to guess a gradient

Backpropagation within neural networks leverages a fundamental element of
automatic differentiation, which is referred to as the reverse mode
differentiation, or vector Jacobian Product (VJP) or, in the context of
differential geometry, known as the pull-back process. The computation of
gradient are important as update of neural network parameters is performed
using gradient descent method. In this study, we present a genric randomized
method, which updates the parameters of neural networks by using directional
derivatives of loss functions computed efficiently by using forward mode AD or
Jacobian vector Product (JVP). These JVP are computed along the random
directions sampled from different probability distributions e.g., Bernoulli,
Normal, Wigner, Laplace and Uniform distributions. The computation of gradient
is performed during the forward pass of the neural network. We also present a
rigorous analysis of the presented methods providing the rate of convergence
along with the computational experiments deployed in scientific Machine
learning in particular physics-informed neural networks and Deep Operator
Networks.

神经网络的反向传播过程利用了自动微分的基本要素，通过前向模式的自动微分或 Jacobian 向量乘积 (JVP) 来计算损失函数的方向导数，并使用随机采样的不同概率分布计算这些方向导数，本文对这些方法进行了严格的分析并给出了收敛速率，同时还进行了在科学机器学习中部署的计算实验，特别是应用于物理信息神经网络和深度算子网络。

用于优化算法的随机前向模式自动微分

Randomized Forward Mode of Automatic Differentiation for Optimization  Algorithms

Backpropagation (BP) is the dominant and most successful method for training
parameters of deep neural network models. However, BP relies on two
computationally distinct phases, does not provide a satisfactory explanation of
biological learning, and can be challenging to apply for training of networks
with discontinuities or noisy node dynamics. By comparison, node perturbation
(NP) proposes learning by the injection of noise into the network activations,
and subsequent measurement of the induced loss change. NP relies on two forward
(inference) passes, does not make use of network derivatives, and has been
proposed as a model for learning in biological systems. However, standard NP is
highly data inefficient and unstable due to its unguided, noise-based, activity
search. In this work, we investigate different formulations of NP and relate it
to the concept of directional derivatives as well as combining it with a
decorrelating mechanism for layer-wise inputs. We find that a closer alignment
with directional derivatives, and induction of decorrelation of inputs at every
layer significantly enhances performance of NP learning making it competitive
with BP.

Backpropagation (BP) is the dominant method for training deep neural networks, but node perturbation (NP) proposes learning through noise injection and measurement of induced loss change, yielding competitive performance when aligned with directional derivatives and combining with a decorrelating mechanism for layer-wise inputs.