We propose to optimize the activation functions of a deep neural network by
adding a corresponding functional regularization to the cost function. We
justify the use of a second-order total-variation criterion. This allows us to
derive a general representer theorem for deep neural networks that makes a
direct connection with splines and sparsity. Specifically, we show that the
optimal network configuration can be achieved with activation functions that
are nonuniform linear splines with adaptive knots. The bottom line is that the
action of each neuron is encoded by a spline whose parameters (including the
number of knots) are optimized during the training procedure. The scheme
results in a computational structure that is compatible with the existing
deep-ReLU, parametric ReLU, APL (adaptive piecewise-linear) and MaxOut
architectures. It also suggests novel optimization challenges, while making the
link with $\ell_1$ minimization and sparsity-promoting techniques explicit.

通过引入功能规则化到损失函数中，基于二阶总变差准则，提出了一种可以用于通信、编码和学习的深度神经网络优化方法；同时，通过使用自适应节点的非均匀线性样条，可以实现每个神经元的动作编码。

深度神经网络的表示定理

A representer theorem for deep neural networks

The ability to discover physical laws and governing equations from data is
one of humankind's greatest intellectual achievements. A quantitative
understanding of dynamic constraints and balances in nature has facilitated
rapid development of knowledge and enabled advanced technological achievements,
including aircraft, combustion engines, satellites, and electrical power. In
this work, we combine sparsity-promoting techniques and machine learning with
nonlinear dynamical systems to discover governing physical equations from
measurement data. The only assumption about the structure of the model is that
there are only a few important terms that govern the dynamics, so that the
equations are sparse in the space of possible functions; this assumption holds
for many physical systems. In particular, we use sparse regression to determine
the fewest terms in the dynamic governing equations required to accurately
represent the data. The resulting models are parsimonious, balancing model
complexity with descriptive ability while avoiding overfitting. We demonstrate
the algorithm on a wide range of problems, from simple canonical systems,
including linear and nonlinear oscillators and the chaotic Lorenz system, to
the fluid vortex shedding behind an obstacle. The fluid example illustrates the
ability of this method to discover the underlying dynamics of a system that
took experts in the community nearly 30 years to resolve. We also show that
this method generalizes to parameterized, time-varying, or externally forced
systems.

利用稀疏度技术和机器学习与非线性动力系统相结合，从测量数据中发现控制物理方程，并使用稀疏回归确定动态控制方程中的最少项以准确地表示数据。