We replace the multiplication and sigmoid function of the conventional
recurrent gate with addition and ReLU activation. This mechanism is designed to
maintain long-term memory for sequence processing but at a reduced
computational cost, thereby opening up for more efficient execution or larger
models on restricted hardware. Recurrent Neural Networks (RNNs) with gating
mechanisms such as LSTM and GRU have been widely successful in learning from
sequential data due to their ability to capture long-term dependencies.
Conventionally, the update based on current inputs and the previous state
history is each multiplied with dynamic weights and combined to compute the
next state. However, multiplication can be computationally expensive,
especially for certain hardware architectures or alternative arithmetic systems
such as homomorphic encryption. It is demonstrated that the novel gating
mechanism can capture long-term dependencies for a standard synthetic sequence
learning task while significantly reducing computational costs such that
execution time is reduced by half on CPU and by one-third under encryption.
Experimental results on handwritten text recognition tasks furthermore show
that the proposed architecture can be trained to achieve comparable accuracy to
conventional GRU and LSTM baselines. The gating mechanism introduced in this
paper may enable privacy-preserving AI applications operating under homomorphic
encryption by avoiding the multiplication of encrypted variables. It can also
support quantization in (unencrypted) plaintext applications, with the
potential for substantial performance gains since the addition-based
formulation can avoid the expansion to double precision often required for
multiplication.

通过替换传统的循环门中的乘法和 Sigmoid 函数为加法和 ReLU 激活函数，该研究提出了一种可以在更低计算成本下维持长期记忆用于序列处理的机制，从而在受限制硬件上实现更高效的执行或更大模型。此机制能够在保持较高的计算效率的同时，捕捉到序列数据的长期依赖关系，并且在 CPU 上减少了一半的执行时间，在加密环境下减少了三分之一的执行时间，实验结果表明该机制与传统的 GRU 和 LSTM 基准模型相比，能够实现可比较的准确性。同时，该机制还能避免加密变量的乘法操作，从而支持具有保护隐私的同态加密人工智能应用，并能在（非加密）明文应用中支持量化操作，潜在地带来显著的性能提升。