As a neural network's depth increases, it can achieve strong generalization performance. Training, however, becomes challenging due to gradient issues. Theoretical research and various methods have been introduced to address this issues. However, research on weight initialization methods that can be effectively applied to tanh neural networks of varying sizes still needs to be completed. This paper presents a novel weight initialization method for Feedforward Neural Networks with tanh activation function. Based on an analysis of the fixed points of the function $\tanh(ax)$, our proposed method aims to determine values of $a$ that prevent the saturation of activations. A series of experiments on various classification datasets demonstrate that the proposed method is more robust to network size variations than the existing method. Furthermore, when applied to Physics-Informed Neural Networks, the method exhibits faster convergence and robustness to variations of the network size compared to Xavier initialization in problems of Partial Differential Equations.

本研究针对双曲正切神经网络在深度增加时训练难度加大的问题，提出了一种新的权重初始化方法。通过分析双曲正切函数的固定点，确定了防止激活饱和的权重值，从而提升了网络在不同规模下的鲁棒性与收敛速度。实验表明，该方法在求解偏微分方程问题时优于现有的Xavier初始化。

具有固定点分析的双曲正切神经网络的鲁棒权重初始化