Consider the problem: given the data pair $(\mathbf{x}, \mathbf{y})$ drawn
from a population with $f_*(x) = \mathbf{E}[\mathbf{y} | \mathbf{x} = x]$,
specify a neural network model and run gradient flow on the weights over time
until reaching any stationarity. How does $f_t$, the function computed by the
neural network at time $t$, relate to $f_*$, in terms of approximation and
representation? What are the provable benefits of the adaptive representation
by neural networks compared to the pre-specified fixed basis representation in
the classical nonparametric literature? We answer the above questions via a
dynamic reproducing kernel Hilbert space (RKHS) approach indexed by the
training process of neural networks. Firstly, we show that when reaching any
local stationarity, gradient flow learns an adaptive RKHS representation and
performs the global least-squares projection onto the adaptive RKHS,
simultaneously. Secondly, we prove that as the RKHS is data-adaptive and
task-specific, the residual for $f_*$ lies in a subspace that is potentially
much smaller than the orthogonal complement of the RKHS. The result formalizes
the representation and approximation benefits of neural networks. Lastly, we
show that the neural network function computed by gradient flow converges to
the kernel ridgeless regression with an adaptive kernel, in the limit of
vanishing regularization. The adaptive kernel viewpoint provides new angles of
studying the approximation, representation, generalization, and optimization
advantages of neural networks.

利用动态再生核赋范空间方法研究神经网络的逼近和表示优势，证明其自适应核函数表示比经典非参数方法中的预先指定的固定基础表示更具优势，同时通过渐进正则化说明其渐进收敛性。