These lecture notes provide an overview of Neural Network architectures from
a mathematical point of view. Especially, Machine Learning with Neural Networks
is seen as an optimization problem. Covered are an introduction to Neural
Networks and the following architectures: Feedforward Neural Network,
Convolutional Neural Network, ResNet, and Recurrent Neural Network.

这篇论文从数学角度概述了神经网络结构，尤其是把神经网络看作是一个优化问题。涵盖了前馈神经网络，卷积神经网络，ResNet 和循环神经网络等结构。

神经网络架构讲义

Lecture Notes: Neural Network Architectures

Wide neural networks with random weights and biases are Gaussian processes,
as originally observed by Neal (1995) and more recently by Lee et al. (2018)
and Matthews et al. (2018) for deep fully-connected networks, as well as by
Novak et al. (2019) and Garriga-Alonso et al. (2019) for deep convolutional
networks. We show that this Neural Network-Gaussian Process correspondence
surprisingly extends to all modern feedforward or recurrent neural networks
composed of multilayer perceptron, RNNs (e.g. LSTMs, GRUs), (nD or graph)
convolution, pooling, skip connection, attention, batch normalization, and/or
layer normalization. More generally, we introduce a language for expressing
neural network computations, and our result encompasses all such expressible
neural networks. This work serves as a tutorial on the *tensor programs*
technique formulated in Yang (2019) and elucidates the Gaussian Process results
obtained there. We provide open-source implementations of the Gaussian Process
kernels of simple RNN, GRU, transformer, and batchnorm+ReLU network at
github.com/thegregyang/GP4A.

本文通过引入一种表达神经网络计算的语言，证明了具有随机权重和偏置的宽深度神经网络是高斯过程。 结果表明，这种神经网络 - 高斯过程对应关系即使对于所有现代前馈或递归神经网络都适用。

张量程序 I：任何架构的宽前馈或递归神经网络都是高斯过程

Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any  Architecture are Gaussian Processes

In this paper, we explore some basic questions on the complexity of training
neural networks with ReLU activation function. We show that it is NP-hard to
train a two-hidden layer feedforward ReLU neural network. If dimension of the
input data and the network topology is fixed, then we show that there exists a
polynomial time algorithm for the same training problem. We also show that if
sufficient over-parameterization is provided in the first hidden layer of ReLU
neural network, then there is a polynomial time algorithm which finds weights
such that output of the over-parameterized ReLU neural network matches with the
output of the given data.

本文探讨了采用 ReLU 激活函数训练神经网络的复杂度问题，研究表明在神经元构架固定的情况下，两层 ReLU 神经网络的训练是 NP - 难问题，但在第一隐藏层提供足够的超参数时，可以通过多项式时间算法找到合适的权重。