State-space models have gained popularity in sequence modelling due to their
simple and efficient network structures. However, the absence of nonlinear
activation along the temporal direction limits the model's capacity. In this
paper, we prove that stacking state-space models with layer-wise nonlinear
activation is sufficient to approximate any continuous sequence-to-sequence
relationship. Our findings demonstrate that the addition of layer-wise
nonlinear activation enhances the model's capacity to learn complex sequence
patterns. Meanwhile, it can be seen both theoretically and empirically that the
state-space models do not fundamentally resolve the exponential decaying memory
issue. Theoretical results are justified by numerical verifications.

通过在时间方向上添加逐层非线性激活函数，我们证明堆叠状态空间模型能够近似任意连续序列关系，并增强模型学习复杂序列模式的能力，同时理论和实证结果表明状态空间模型并不能根本解决指数衰减记忆问题。

具有逐层非线性的状态空间模型是具有指数衰减记忆力的通用逼近器

State-space Models with Layer-wise Nonlinearity are Universal  Approximators with Exponential Decaying Memory

Hypergraphs provide a natural representation for many real world datasets. We
propose a novel framework, HNHN, for hypergraph representation learning. HNHN
is a hypergraph convolution network with nonlinear activation functions applied
to both hypernodes and hyperedges, combined with a normalization scheme that
can flexibly adjust the importance of high-cardinality hyperedges and
high-degree vertices depending on the dataset. We demonstrate improved
performance of HNHN in both classification accuracy and speed on real world
datasets when compared to state of the art methods.

该论文提出了一种基于超图卷积网络的新型框架 HNHN 来进行超图表示学习，其中包括非线性激活函数和灵活的归一化方案，实验结果表明与现有方法相比，在真实世界数据集上表现更好，具有更高的分类精度和速度。

HNHN: 具有超边神经元的超图网络

HNHN: Hypergraph Networks with Hyperedge Neurons

Can multilayer neural networks -- typically constructed as highly complex
structures with many nonlinearly activated neurons across layers -- behave in a
non-trivial way that yet simplifies away a major part of their complexities? In
this work, we uncover a phenomenon in which the behavior of these complex
networks -- under suitable scalings and stochastic gradient descent dynamics --
becomes independent of the number of neurons as this number grows sufficiently
large. We develop a formalism in which this many-neurons limiting behavior is
captured by a set of equations, thereby exposing a previously unknown operating
regime of these networks. While the current pursuit is mathematically
non-rigorous, it is complemented with several experiments that validate the
existence of this behavior.

本文发现随着神经元数量的增加，经过适当的缩放和随机梯度下降动力学，多层神经网络的行为变得独立于神经元数量，发展了一个形式体系来捕捉这种多神经元限制行为，相关实验证实了这种独立性的存在。

多层神经网络学习动力学的平均场极限

Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks

This work attempts to address two fundamental questions about the structure
of the convolutional neural networks (CNN): 1) why a non-linear activation
function is essential at the filter output of every convolutional layer? 2)
what is the advantage of the two-layer cascade system over the one-layer
system? A mathematical model called the "REctified-COrrelations on a Sphere"
(RECOS) is proposed to answer these two questions. After the CNN training
process, the converged filter weights define a set of anchor vectors in the
RECOS model. Anchor vectors represent the frequently occurring patterns (or the
spectral components). The necessity of rectification is explained using the
RECOS model. Then, the behavior of a two-layer RECOS system is analyzed and
compared with its one-layer counterpart. The LeNet-5 and the MNIST dataset are
used to illustrate discussion points. Finally, the RECOS model is generalized
to a multi-layer system with the AlexNet as an example.
Keywords: Convolutional Neural Network (CNN), Nonlinear Activation, RECOS
Model, Rectified Linear Unit (ReLU), MNIST Dataset.

本研究利用 RECOS 模型解释了卷积神经网络（CNN）的非线性激活函数的必要性，探讨了两层级联系统相对于一层的优势，并将 RECOS 模型推广到了多层系统中。