Activation functions are one of the key components of a neural network. The
most commonly used activation functions can be classed into the category of
continuously differentiable (e.g. tanh) and linear-unit functions (e.g. ReLU),
both having their own strengths and drawbacks with respect to downstream
performance and representation capacity through learning (e.g. measured by the
number of dead neurons and the effective rank). In reinforcement learning, the
performance of continuously differentiable activations often falls short as
compared to linear-unit functions. From the perspective of the activations in
the last hidden layer, this paper provides insights regarding this
sub-optimality and explores how activation functions influence the occurrence
of dead neurons and the magnitude of the effective rank. Additionally, a novel
neural architecture is proposed that leverages the product of independent
activation values. In the Atari domain, we show faster learning, a reduction in
dead neurons and increased effective rank.

通过研究激活函数对死神经元和有效秩大小的影响，本文提出了一个新的神经网络结构，并展示了在 Atari 领域中学习速度更快、死神经元减少和有效秩增加的结果。