Activation functions are essential to introduce nonlinearity into neural
networks, with the Rectified Linear Unit (ReLU) often favored for its
simplicity and effectiveness. Motivated by the structural similarity between a
shallow Feedforward Neural Network (FNN) and a single iteration of the
Projected Gradient Descent (PGD) algorithm, a standard approach for solving
constrained optimization problems, we consider ReLU as a projection from R onto
the nonnegative half-line R+. Building on this interpretation, we extend ReLU
by substituting it with a generalized projection operator onto a convex cone,
such as the Second-Order Cone (SOC) projection, thereby naturally extending it
to a Multivariate Projection Unit (MPU), an activation function with multiple
inputs and multiple outputs. We further provide a mathematical proof
establishing that FNNs activated by SOC projections outperform those utilizing
ReLU in terms of expressive power. Experimental evaluations on widely-adopted
architectures further corroborate MPU's effectiveness against a broader range
of existing activation functions.

基于结构相似性，通过将 ReLU 替换为广义投影算子，将其扩展为具有多个输入和多个输出的多元投影单元 (MPU)，证明了在表达能力方面，由 SOC 投影激活的 FNN 优于利用 ReLU 的 FNN。实验评估进一步证实了 MPU 在更广泛的现有激活函数范围内的有效性。