Randomized algorithms exploit stochasticity to reduce computational
complexity. One important example is random feature regression (RFR) that
accelerates Gaussian process regression (GPR). RFR approximates an unknown
function with a random neural network whose hidden weights and biases are
sampled from a probability distribution. Only the final output layer is fit to
data. In randomized algorithms like RFR, the hyperparameters that characterize
the sampling distribution greatly impact performance, yet are not directly
accessible from samples. This makes optimization of hyperparameters via
standard (gradient-based) optimization tools inapplicable. Inspired by Bayesian
ideas from GPR, this paper introduces a random objective function that is
tailored for hyperparameter tuning of vector-valued random features. The
objective is minimized with ensemble Kalman inversion (EKI). EKI is a
gradient-free particle-based optimizer that is scalable to high-dimensions and
robust to randomness in objective functions. A numerical study showcases the
new black-box methodology to learn hyperparameter distributions in several
problems that are sensitive to the hyperparameter selection: two global
sensitivity analyses, integrating a chaotic dynamical system, and solving a
Bayesian inverse problem from atmospheric dynamics. The success of the proposed
EKI-based algorithm for RFR suggests its potential for automated optimization
of hyperparameters arising in other randomized algorithms.

基于随机算法和集合卡尔曼反演的随机特征回归 (RFR) 的随机目标函数为矢量值随机特征的超参数优化引入了一种新的黑盒方法，该方法在多个对超参数选择敏感的问题中展示了成功，包括两个全局敏感性分析、在混沌动力学系统中进行集成和从大气动力学解决贝叶斯反问题。该方法为 RFR 提供了自动优化超参数的潜力。

随机算法的超参数优化：以随机特征为例的案例研究

Hyperparameter Optimization for Randomized Algorithms: A Case Study for  Random Features

In our era of enormous neural networks, empirical progress has been driven by
the philosophy that more is better. Recent deep learning practice has found
repeatedly that larger model size, more data, and more computation (resulting
in lower training loss) improves performance. In this paper, we give
theoretical backing to these empirical observations by showing that these three
properties hold in random feature (RF) regression, a class of models equivalent
to shallow networks with only the last layer trained.
Concretely, we first show that the test risk of RF regression decreases
monotonically with both the number of features and the number of samples,
provided the ridge penalty is tuned optimally. In particular, this implies that
infinite width RF architectures are preferable to those of any finite width. We
then proceed to demonstrate that, for a large class of tasks characterized by
powerlaw eigenstructure, training to near-zero training loss is obligatory:
near-optimal performance can only be achieved when the training error is much
smaller than the test error. Grounding our theory in real-world data, we find
empirically that standard computer vision tasks with convolutional neural
tangent kernels clearly fall into this class. Taken together, our results tell
a simple, testable story of the benefits of overparameterization, overfitting,
and more data in random feature models.

我们的研究通过理论分析和实证结果说明，更大的神经网络模型、更多的数据和更多的计算可以提升随机特征模型的性能。

现代机器学习中，更多才是更好：当无限过参数化时是最优的，而过拟合则是必然的

More is Better in Modern Machine Learning: when Infinite  Overparameterization is Optimal and Overfitting is Obligatory

One of the distinguishing characteristics of modern deep learning systems is
that they typically employ neural network architectures that utilize enormous
numbers of parameters, often in the millions and sometimes even in the
billions. While this paradigm has inspired significant research on the
properties of large networks, relatively little work has been devoted to the
fact that these networks are often used to model large complex datasets, which
may themselves contain millions or even billions of constraints. In this work,
we focus on this high-dimensional regime in which both the dataset size and the
number of features tend to infinity. We analyze the performance of random
feature regression with features $F=f(WX+B)$ for a random weight matrix $W$ and
random bias vector $B$, obtaining exact formulae for the asymptotic training
and test errors for data generated by a linear teacher model. The role of the
bias can be understood as parameterizing a distribution over activation
functions, and our analysis directly generalizes to such distributions, even
those not expressible with a traditional additive bias. Intriguingly, we find
that a mixture of nonlinearities can improve both the training and test errors
over the best single nonlinearity, suggesting that mixtures of nonlinearities
might be useful for approximate kernel methods or neural network architecture
design.

研究分析使用大型神经网络架构和随机特征回归的深度学习在高维数据集的性能表现，并发现混合非线性函数可提高训练与测试性能，为近似核方法或神经网络架构设计提供了参考。