It is generally thought that the use of stochastic activation functions in
deep learning architectures yield models with superior generalization
abilities. However, a sufficiently rigorous statement and theoretical proof of
this heuristic is lacking in the literature. In this paper, we provide several
novel contributions to the literature in this regard. Defining a new notion of
nonlocal directional derivative, we analyze its theoretical properties
(existence and convergence). Second, using a probabilistic reformulation, we
show that nonlocal derivatives are epsilon-sub gradients, and derive sample
complexity results for convergence of stochastic gradient descent-like methods
using nonlocal derivatives. Finally, using our analysis of the nonlocal
gradient of Holder continuous functions, we observe that sample paths of
Brownian motion admit nonlocal directional derivatives, and the nonlocal
derivatives of Brownian motion are seen to be Gaussian processes with
computable mean and standard deviation. Using the theory of nonlocal
directional derivatives, we solve a highly nondifferentiable and nonconvex
model problem of parameter estimation on image articulation manifolds. Using
Brownian motion infused ReLU activation functions with the nonlocal gradient in
place of the usual gradient during backpropagation, we also perform experiments
on multiple well-studied deep learning architectures. Our experiments indicate
the superior generalization capabilities of Brownian neural activation
functions in low-training data regimes, where the use of stochastic neurons
beats the deterministic ReLU counterpart.

在本文中，我们通过定义非局部方向导数的新概念，并分析其理论属性，推导出非局部导数是 ε-sub 梯度的概率重整结果，以及收敛于随机梯度下降方法的样本复杂度结果。最终，利用非局部梯度解决了图像关节流形上参数估计的高度不可微和非凸模型问题，并通过实验表明了在低训练数据情况下，使用布朗运动灌注的 ReLu 激活函数和非局部梯度胜过确定性的 ReLu 激活函数的优越的泛化能力。