Appropriate weight initialization settings, along with the ReLU activation
function, have been a cornerstone of modern deep learning, making it possible
to train and deploy highly effective and efficient neural network models across
diverse artificial intelligence. The problem of dying ReLU, where ReLU neurons
become inactive and yield zero output, presents a significant challenge in the
training of deep neural networks with ReLU activation function. Theoretical
research and various methods have been introduced to address the problem.
However, even with these methods and research, training remains challenging for
extremely deep and narrow feedforward networks with ReLU activation function.
In this paper, we propose a new weight initialization method to address this
issue. We prove the properties of the proposed initial weight matrix and
demonstrate how these properties facilitate the effective propagation of signal
vectors. Through a series of experiments and comparisons with existing methods,
we demonstrate the effectiveness of the new initialization method.

通过引入新的权重初始化方法，本论文证明了所提出的初始权重矩阵的特性，展示了这些特性如何促进信号向量的有效传播，并通过一系列实验和与现有方法的比较展示了新的初始化方法的有效性。

深且窄的前馈神经网络的改进权重初始化

Improved weight initialization for deep and narrow feedforward neural  network

Activation functions (AFs) are crucial components of deep neural networks
(DNNs), having a significant impact on their performance. An activation
function in a DNN is typically a smooth, nonlinear function that transforms an
input signal into an output signal for the subsequent layer. In this paper, we
propose the Parametric Leaky Tanh (PLTanh), a novel hybrid activation function
designed to combine the strengths of both the Tanh and Leaky ReLU (LReLU)
activation functions. PLTanh is differentiable at all points and addresses the
'dying ReLU' problem by ensuring a non-zero gradient for negative inputs,
consistent with the behavior of LReLU. By integrating the unique advantages of
these two diverse activation functions, PLTanh facilitates the learning of more
intricate nonlinear relationships within the network. This paper presents an
empirical evaluation of PLTanh against established activation functions, namely
ReLU, LReLU, and ALReLU utilizing five diverse datasets.

本文在深度神经网络中提出了 Parametric Leaky Tanh（PLTanh）作为一种新的混合激活函数，结合了 Tanh 和 Leaky ReLU 激活函数的优势，通过保证负输入的梯度非零，解决了 'dying ReLU' 问题，并通过与 ReLU、LReLU 和 ALReLU 在五个不同数据集上的实证评估，展示了 PLTanh 的优势。