We study the supervised learning problem under either of the following two
models: (1) Feature vectors ${\boldsymbol x}_i$ are $d$-dimensional Gaussians
and responses are $y_i = f_*({\boldsymbol x}_i)$ for $f_*$ an unknown quadratic
function; (2) Feature vectors ${\boldsymbol x}_i$ are distributed as a mixture
of two $d$-dimensional centered Gaussians, and $y_i$'s are the corresponding
class labels. We use two-layers neural networks with quadratic activations, and
compare three different learning regimes: the random features (RF) regime in
which we only train the second-layer weights; the neural tangent (NT) regime in
which we train a linearization of the neural network around its initialization;
the fully trained neural network (NN) regime in which we train all the weights
in the network. We prove that, even for the simple quadratic model of point
(1), there is a potentially unbounded gap between the prediction risk achieved
in these three training regimes, when the number of neurons is smaller than the
ambient dimension. When the number of neurons is larger than the number of
dimensions, the problem is significantly easier and both NT and NN learning
achieve zero risk.

在两种模型下，我们使用二层神经网络带有二次激活函数进行监督学习，证明了当神经元数量小于特征向量维度时，在随机特征和神经切线等三种不同的学习方案中，性能存在无限制的差距。当神经元数量大于特征向量维度时，这个问题变得容易，神经切线和完全训练的神经网络都能实现零风险。

二层神经网络懒惰训练的局限性

Limitations of Lazy Training of Two-layers Neural Networks

The reparameterization trick is widely used in variational inference as it
yields more accurate estimates of the gradient of the variational objective
than alternative approaches such as the score function method. Although there
is overwhelming empirical evidence in the literature showing its success, there
is relatively little research exploring why the reparameterization trick is so
effective. We explore this under the idealized assumptions that the variational
approximation is a mean-field Gaussian density and that the log of the joint
density of the model parameters and the data is a quadratic function that
depends on the variational mean. From this, we show that the marginal variances
of the reparameterization gradient estimator are smaller than those of the
score function gradient estimator. We apply the result of our idealized
analysis to real-world examples.

本文探讨变分推断问题中使用重新参数化技巧的效果，通过理论分析和实验展示了其优越性，重点在于阐述它相比于其他方法更准确地估计了 variational objective 的梯度。

重参数化技巧的方差降低特性

Variance reduction properties of the reparameterization trick

We consider the fundamental problem of maximizing a general quadratic
function over an ellipsoidal domain, also known as the trust region problem. We
give the first provable linear-time (in the number of non-zero entries of the
input) algorithm for approximately solving this problem. Specifically, our
algorithm returns an $\epsilon$-approximate solution in time
$\tilde{O}(N/\sqrt{\epsilon})$, where $N$ is the number of non-zero entries in
the input. This matches the runtime of Nesterov's accelerated gradient descent,
suitable for the special case in which the quadratic function is concave, and
the runtime of the Lanczos method which is applicable when the problem is
purely quadratic.

本文提出了一种在椭圆域上求解一般二次函数最大化问题的近似算法，时间复杂度为线性，能获得 ϵ- 近似解，与 Nesterov 的加速梯度下降算法和 Lanczos 方法的运行时相匹配。