We investigate the variational optimality (specifically, the Banach space optimality) of a large class of neural architectures with multivariate nonlinearities/activation functions. To that end, we construct a new family of Banach spaces defined via a regularization operator and the $k$-plane transform. We prove a representer theorem that states that the solution sets to learning problems posed over these Banach spaces are completely characterized by neural architectures with multivariate nonlinearities. These optimal architectures have skip connections and are tightly connected to orthogonal weight normalization and multi-index models, both of which have received considerable interest in the neural network community. Our framework is compatible with a number of classical nonlinearities including the rectified linear unit (ReLU) activation function, the norm activation function, and the radial basis functions found in the theory of thin-plate/polyharmonic splines. We also show that the underlying spaces are special instances of reproducing kernel Banach spaces and variation spaces. Our results shed light on the regularity of functions learned by neural networks trained on data, particularly with multivariate nonlinearities, and provide new theoretical motivation for several architectural choices found in practice.

通过构建一类新的经过正则化操作与$k$-plane变换定义的Banach空间，并证明具有多元非线性的神经结构是这些Banach空间中学习问题解集的完全刻画，我们研究了一大类神经结构的变分最优性（具体而言指Banach空间的最优性）。这些最优的神经结构具有跳跃连接，与正交权重归一化和多索引模型紧密相关，这在神经网络领域引起了相当大的关注。此外，我们还展示了底层空间是再生核Banach空间和变分空间的特殊实例，并为神经网络在数据上学习的函数的正则性提供了新的理论动机，尤其是在具有多元非线性时，并提供了对实践中一些架构选择的新的理论动力。

具有多元非线性的神经结构的巴拿赫空间优化性