Neural networks have been extensively applied to a variety of tasks,
achieving astounding results. Applying neural networks in the scientific field
is an important research direction that is gaining increasing attention. In
scientific applications, the scale of neural networks is generally
moderate-size, mainly to ensure the speed of inference during application.
Additionally, comparing neural networks to traditional algorithms in scientific
applications is inevitable. These applications often require rapid
computations, making the reduction of neural network sizes increasingly
important. Existing work has found that the powerful capabilities of neural
networks are primarily due to their non-linearity. Theoretical work has
discovered that under strong non-linearity, neurons in the same layer tend to
behave similarly, a phenomenon known as condensation. Condensation offers an
opportunity to reduce the scale of neural networks to a smaller subnetwork with
similar performance. In this article, we propose a condensation reduction
algorithm to verify the feasibility of this idea in practical problems. Our
reduction method can currently be applied to both fully connected networks and
convolutional networks, achieving positive results. In complex combustion
acceleration tasks, we reduced the size of the neural network to 41.7% of its
original scale while maintaining prediction accuracy. In the CIFAR10 image
classification task, we reduced the network size to 11.5% of the original
scale, still maintaining a satisfactory validation accuracy. Our method can be
applied to most trained neural networks, reducing computational pressure and
improving inference speed.

我们提出了一种凝结缩减算法，验证了这一想法在实际问题中的可行性，可以将神经网络的规模减小，并在保持预测准确性的同时减少计算压力和提高推理速度。

压缩中等规模深度神经网络的高效灵活方法

Efficient and Flexible Method for Reducing Moderate-size Deep Neural  Networks with Condensation

Empirical and theoretical works show that the input weights of two-layer
neural networks, when initialized with small values, converge towards isolated
orientations. This phenomenon, referred to as condensation, indicates that the
gradient descent methods tend to spontaneously reduce the complexity of neural
networks during the training process. In this work, we elucidate the mechanisms
behind the condensation phenomena occurring in the training of three-layer
neural networks and distinguish it from the training of two-layer neural
networks. Through rigorous theoretical analysis, we establish the blow-up
property of effective dynamics and present a sufficient condition for the
occurrence of condensation, findings that are substantiated by experimental
results. Additionally, we explore the association between condensation and the
low-rank bias observed in deep matrix factorization.

通过理论分析，我们揭示了三层神经网络训练中凝聚现象的机制，并从二层神经网络训练中进行了区分。我们还建立了有效动力学的爆炸特性，并给出了凝聚现象发生的充分条件，并通过实验证实了这些发现。此外，我们探索了凝聚与深度矩阵分解中观察到的低秩偏差之间的关联。

三层神经网络动力学：初始凝聚

On the dynamics of three-layer neural networks: initial condensation

Previous research has shown that fully-connected networks with small
initialization and gradient-based training methods exhibit a phenomenon known
as condensation during training. This phenomenon refers to the input weights of
hidden neurons condensing into isolated orientations during training, revealing
an implicit bias towards simple solutions in the parameter space. However, the
impact of neural network structure on condensation has not been investigated
yet. In this study, we focus on the investigation of convolutional neural
networks (CNNs). Our experiments suggest that when subjected to small
initialization and gradient-based training methods, kernel weights within the
same CNN layer also cluster together during training, demonstrating a
significant degree of condensation. Theoretically, we demonstrate that in a
finite training period, kernels of a two-layer CNN with small initialization
will converge to one or a few directions. This work represents a step towards a
better understanding of the non-linear training behavior exhibited by neural
networks with specialized structures.

本研究探讨了卷积神经网络在小初始化和梯度训练方法下内核权重的凝聚现象，实验证明该现象在卷积神经网络中同样存在且显著。理论上，本研究证明在有限的训练期间，具有小初始化的两层卷积神经网络内核将收敛至一个或几个方向，为对具有专业结构的神经网络表现出的非线性训练行为的更好理解迈出了一步。

卷积神经网络初步凝结的理解

Understanding the Initial Condensation of Convolutional Neural Networks

Condensation phenomena arise through a collective behaviour of particles.
They are observed in both classical and quantum systems, ranging from the
formation of traffic jams in mass transport models to the macroscopic
occupation of the energetic ground state in ultra-cold bosonic gases
(Bose-Einstein condensation). Recently, it has been shown that a driven and
dissipative system of bosons may form multiple condensates. Which states become
the condensates has, however, remained elusive thus far. The dynamics of this
condensation are described by coupled birth-death processes, which also occur
in evolutionary game theory. Here, we apply concepts from evolutionary game
theory to explain the formation of multiple condensates in such
driven-dissipative bosonic systems. We show that vanishing of relative entropy
production determines their selection. The condensation proceeds exponentially
fast, but the system never comes to rest. Instead, the occupation numbers of
condensates may oscillate, as we demonstrate for a rock-paper-scissors game of
condensates.

通过进化博弈理论的概念，我们解释了驱动耗散波色子系统中多重冷凝的形成，结果表明相对熵产生的消失决定了它们的选择。