Recurrent neural networks (RNNs) are capable of modeling the temporal
dynamics of complex sequential information. However, the structures of existing
RNN neurons mainly focus on controlling the contributions of current and
historical information but do not explore the different importance levels of
different elements in an input vector of a time slot. We propose adding a
simple yet effective Element-wiseAttention Gate (EleAttG) to an RNN block
(e.g., all RNN neurons in a network layer) that empowers the RNN neurons to
have the attentiveness capability. For an RNN block, an EleAttG is added to
adaptively modulate the input by assigning different levels of importance,
i.e., attention, to each element/dimension of the input. We refer to an RNN
block equipped with an EleAttG as an EleAtt-RNN block. Specifically, the
modulation of the input is content adaptive and is performed at fine
granularity, being element-wise rather than input-wise. The proposed EleAttG,
as an additional fundamental unit, is general and can be applied to any RNN
structures, e.g., standard RNN, Long Short-Term Memory (LSTM), or Gated
Recurrent Unit (GRU). We demonstrate the effectiveness of the proposed
EleAtt-RNN by applying it to the action recognition tasks on both 3D human
skeleton data and RGB videos. Experiments show that adding attentiveness
through EleAttGs to RNN blocks significantly boosts the power of RNNs.

本文提出在 RNN block 中添加 Element-wiseAttention Gate 来实现注意力功能，通过元素级而不是输入级的自适应调节，从而提高 RNN 的效能，在人体动作识别任务中进行了实验并验证了该方法的有效性。