In the past, research on a single low dimensional activation function in
networks has led to internal covariate shift and gradient deviation problems. A
relatively small research area is how to use function combinations to provide
property completion for a single activation function application. We propose a
network adversarial method to address the aforementioned challenges. This is
the first method to use different activation functions in a network. Based on
the existing activation functions in the current network, an adversarial
function with opposite derivative image properties is constructed, and the two
are alternately used as activation functions for different network layers. For
complex situations, we propose a method of high-dimensional function graph
decomposition(HD-FGD), which divides it into different parts and then passes
through a linear layer. After integrating the inverse of the partial
derivatives of each decomposed term, we obtain its adversarial function by
referring to the computational rules of the decomposition process. The use of
network adversarial methods or the use of HD-FGD alone can effectively replace
the traditional MLP+activation function mode. Through the above methods, we
have achieved a substantial improvement over standard activation functions
regarding both training efficiency and predictive accuracy. The article
addresses the adversarial issues associated with several prevalent activation
functions, presenting alternatives that can be seamlessly integrated into
existing models without any adverse effects. We will release the code as open
source after the conference review process is completed.

通过网络对抗方法，使用不同的激活函数解决传统 MLP + 激活函数模式中的内在双稳问题，提高训练效率和预测准确率。