The training of neural networks requires tedious and often manual tuning of the network architecture. We propose a systematic method to insert new layers during the training process, which eliminates the need to choose a fixed network size before training. Our technique borrows techniques from constrained optimization and is based on first-order sensitivity information of the objective with respect to the virtual parameters that additional layers, if inserted, would offer. We consider fully connected feedforward networks with selected activation functions as well as residual neural networks. In numerical experiments, the proposed sensitivity-based layer insertion technique exhibits improved training decay, compared to not inserting the layer. Furthermore, the computational effort is reduced in comparison to inserting the layer from the beginning. The code is available at \url{https://github.com/LeonieKreis/layer_insertion_sensitivity_based}.

我们提出了一种系统的方法，在训练过程中插入新层，从而消除了在训练前选择固定网络大小的需要。我们的技术借鉴了约束优化技术，并且基于目标函数对虚拟参数的一阶敏感性信息，此虚拟参数代表了如果插入额外层将会提供的信息。通过数值实验，我们的敏感性层插入技术相较于不插入层而言，展现出更好的训练性能衰减，并大大降低了计算复杂度。

基于灵敏度的残差和前馈神经网络层插入