In the literature on deep neural networks, there is considerable interest in
developing activation functions that can enhance neural network performance. In
recent years, there has been renewed scientific interest in proposing
activation functions that can be trained throughout the learning process, as
they appear to improve network performance, especially by reducing overfitting.
In this paper, we propose a trainable activation function whose parameters need
to be estimated. A fully Bayesian model is developed to automatically estimate
from the learning data both the model weights and activation function
parameters. An MCMC-based optimization scheme is developed to build the
inference. The proposed method aims to solve the aforementioned problems and
improve convergence time by using an efficient sampling scheme that guarantees
convergence to the global maximum. The proposed scheme is tested on three
datasets with three different CNNs. Promising results demonstrate the
usefulness of our proposed approach in improving model accuracy due to the
proposed activation function and Bayesian estimation of the parameters.

本文提出了一种可训练的激活函数，使用贝叶斯建模自动估计学习数据中的模型权重和激活函数参数，并使用基于 MCMC 的优化方案进行推断，通过使用有效的采样方案来提高收敛速度并减少过度拟合问题，取得了良好的检验结果，并通过激活函数提高了模型的准确性。

可训练激活函数的稀疏神经网络贝叶斯优化

Bayesian optimization for sparse neural networks with trainable  activation functions

Random Fourier features is one of the most popular techniques for scaling up
kernel methods, such as kernel ridge regression. However, despite impressive
empirical results, the statistical properties of random Fourier features are
still not well understood. In this paper we take steps toward filling this gap.
Specifically, we approach random Fourier features from a spectral matrix
approximation point of view, give tight bounds on the number of Fourier
features required to achieve a spectral approximation, and show how spectral
matrix approximation bounds imply statistical guarantees for kernel ridge
regression.
Qualitatively, our results are twofold: on the one hand, we show that random
Fourier feature approximation can provably speed up kernel ridge regression
under reasonable assumptions. At the same time, we show that the method is
suboptimal, and sampling from a modified distribution in Fourier space, given
by the leverage function of the kernel, yields provably better performance. We
study this optimal sampling distribution for the Gaussian kernel, achieving a
nearly complete characterization for the case of low-dimensional bounded
datasets. Based on this characterization, we propose an efficient sampling
scheme with guarantees superior to random Fourier features in this regime.

本文通过研究谱矩阵近似的角度，给出了随机傅里叶特征的数量界和核岭回归的统计保障，而从核的杠杆函数中改进傅里叶空间的分布采样可获得提高的性能与更优的采样方案。