Neural growth is the process of growing a small neural network to a large
network and has been utilized to accelerate the training of deep neural
networks. One crucial aspect of neural growth is determining the optimal growth
timing. However, few studies investigate this systematically. Our study reveals
that neural growth inherently exhibits a regularization effect, whose intensity
is influenced by the chosen policy for growth timing. While this regularization
effect may mitigate the overfitting risk of the model, it may lead to a notable
accuracy drop when the model underfits. Yet, current approaches have not
addressed this issue due to their lack of consideration of the regularization
effect from neural growth. Motivated by these findings, we propose an
under/over fitting risk-aware growth timing policy, which automatically adjusts
the growth timing informed by the level of potential under/overfitting risks to
address both risks. Comprehensive experiments conducted using CIFAR-10/100 and
ImageNet datasets show that the proposed policy achieves accuracy improvements
of up to 1.3% in models prone to underfitting while achieving similar
accuracies in models suffering from overfitting compared to the existing
methods.

神经生长是将小型神经网络扩展到大型网络的过程，其优化生长时机的研究相对较少。本研究揭示了神经生长固有地具有正则化效应，其强度受生长时机政策的影响。为解决欠拟合和过拟合风险，本文提出了一种基于风险感知的生长时机策略，在潜在的欠拟合和过拟合风险水平的指导下自动调整生长时机。在使用 CIFAR-10/100 和 ImageNet 数据集进行的综合实验中，所提出的策略在容易欠拟合的模型中实现了高达 1.3% 的准确率改善，同时相对于现有方法，在过拟合模型中实现了相似的准确率。

何时生长？深度神经网络中层生长的风险感知策略

When To Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep  Neural Networks

Biological nervous systems are created in a fundamentally different way than
current artificial neural networks. Despite its impressive results in a variety
of different domains, deep learning often requires considerable engineering
effort to design high-performing neural architectures. By contrast, biological
nervous systems are grown through a dynamic self-organizing process. In this
paper, we take initial steps toward neural networks that grow through a
developmental process that mirrors key properties of embryonic development in
biological organisms. The growth process is guided by another neural network,
which we call a Neural Developmental Program (NDP) and which operates through
local communication alone. We investigate the role of neural growth on
different machine learning benchmarks and different optimization methods
(evolutionary training, online RL, offline RL, and supervised learning).
Additionally, we highlight future research directions and opportunities enabled
by having self-organization driving the growth of neural networks.

本文介绍一种神经网络成长的过程，通过一种类似于生物发育的过程，使用神经发育程序 (NDP) 作为一种引导机制，进而自我组织神经网络。作者还探讨了这种方法在不同优化方法（进化训练，在线 RL，离线 RL 和监督学习）和不同机器学习基准上的实验表现，并强调了自组织驱动神经网络成长所带来的未来研究方向和机会。