Recurrent neural networks are increasing popular models for sequential learning. Unfortunately, although the most effective RNN architectures are perhaps excessively complicated, extensive searches have not found simpler alternatives. This paper imports ideas from physics and functional programming into RNN design to provide guiding principles. From physics we introduce type constraints, analogous to the constraints that disqualify adding meters to seconds in physics. From functional programming, we require that strongly-typed architectures factorize into stateless learnware and state-dependent firmware, thereby ameliorating the impact of side-effects. The features learned by strongly-typed nets have a simple semantic interpretation via dynamic average-pooling on one-dimensional convolutions. We also show that strongly-typed gradients are better behaved than in classical architectures, and characterize the representational power of strongly-typed nets. Finally, experiments show that, despite being more constrained, strongly-typed architectures achieve lower training error and comparable generalization error to classical architectures.

本文从物理学和函数式编程中导入思想探讨了基于强类型的循环神经网络（RNN）模型设计问题，提出动态平均池化和一维卷积的简化语义解释。实验证明，虽然更加受限制，但基于强类型的RNN模型在训练时可以达到比传统模型更低的误差率和相当的泛化能力。

强类型递归神经网络