In practice, deeper networks tend to be more powerful than shallow ones, but
this has not been understood theoretically. In this paper, we find the
analytical solution of a three-layer network with a matrix exponential
activation function, i.e., $$ f(X)=W_3\exp(W_2\exp(W_1X)), X\in
\mathbb{C}^{d\times d} $$ have analytical solutions for the equations $$
Y_1=f(X_1),Y_2=f(X_2) $$ for $X_1,X_2,Y_1,Y_2$ with only invertible
assumptions. Our proof shows the power of depth and the use of a non-linear
activation function, since one layer network can only solve one
equation,i.e.,$Y=WX$.

本文通过找到具有矩阵指数激活函数的三层网络的解析解，证明了深度和非线性激活函数的能力，表明深层网络比浅层网络更强大，且一层网络只能解决一个方程。

具有矩阵指数激活函数的三层网络的解析解

Analytical Solution of a Three-layer Network with a Matrix Exponential  Activation Function

The performance of deep network learning strongly depends on the choice of
the non-linear activation function associated with each neuron. However,
deciding on the best activation is non-trivial, and the choice depends on the
architecture, hyper-parameters, and even on the dataset. Typically these
activations are fixed by hand before training. Here, we demonstrate how to
eliminate the reliance on first picking fixed activation functions by using
flexible parametric rational functions instead. The resulting Pad\'e Activation
Units (PAUs) can both approximate common activation functions and also learn
new ones while providing compact representations. Our empirical evidence shows
that end-to-end learning deep networks with PAUs can increase the predictive
performance. Moreover, PAUs pave the way to approximations with provable
robustness. this https URL

本文中介绍了一种新的非线性激活函数 ——Padé 激活单元 (PAUs)，能够代替传统手动选择的固定激活函数，增加深度神经网络的预测性能和可靠性。

Padé 激活单元：在深度网络中端到端学习灵活的激活函数

Padé Activation Units: End-to-end Learning of Flexible Activation  Functions in Deep Networks

The goal of a recommendation system is to predict the interest of a user in a
given item by exploiting the existing set of ratings as well as certain
user/item features. A standard approach to modeling this problem is Inductive
Matrix Completion where the predicted rating is modeled as an inner product of
the user and the item features projected onto a latent space. In order to learn
the parameters effectively from a small number of observed ratings, the latent
space is constrained to be low-dimensional which implies that the parameter
matrix is constrained to be low-rank. However, such bilinear modeling of the
ratings can be limiting in practice and non-linear prediction functions can
lead to significant improvements. A natural approach to introducing
non-linearity in the prediction function is to apply a non-linear activation
function on top of the projected user/item features. Imposition of
non-linearities further complicates an already challenging problem that has two
sources of non-convexity: a) low-rank structure of the parameter matrix, and b)
non-linear activation function. We show that one can still solve the non-linear
Inductive Matrix Completion problem using gradient descent type methods as long
as the solution is initialized well. That is, close to the optima, the
optimization function is strongly convex and hence admits standard optimization
techniques, at least for certain activation functions, such as Sigmoid and
tanh. We also highlight the importance of the activation function and show how
ReLU can behave significantly differently than say a sigmoid function. Finally,
we apply our proposed technique to recommendation systems and semi-supervised
clustering, and show that our method can lead to much better performance than
standard linear Inductive Matrix Completion methods.

本文介绍了在推荐系统和半监督聚类中，如何应用非线性激活函数并使用梯度下降等优化技术，来解决非线性归纳矩阵完成问题，并展示了这种方法的性能优于传统的线性方法。