Recurrent neural networks are a widely used class of neural architectures.
They have, however, two shortcomings. First, it is difficult to understand what
exactly they learn. Second, they tend to work poorly on sequences requiring
long-term memorization, despite having this capacity in principle. We aim to
address both shortcomings with a class of recurrent networks that use a
stochastic state transition mechanism between cell applications. This
mechanism, which we term state-regularization, makes RNNs transition between a
finite set of learnable states. We evaluate state-regularized RNNs on (1)
regular languages for the purpose of automata extraction; (2) nonregular
languages such as balanced parentheses, palindromes, and the copy task where
external memory is required; and (3) real-word sequence learning tasks for
sentiment analysis, visual object recognition, and language modeling. We show
that state-regularization (a) simplifies the extraction of finite state
automata modeling an RNN's state transition dynamics; (b) forces RNNs to
operate more like automata with external memory and less like finite state
machines; (c) makes RNNs have better interpretability and explainability.

通过使用一种被称为状态规范化的机制来处理以前递归神经网络（RNNs）的不足，从而提高 RNNs 的状态转移动态分析和解释性，并将其应用于自动机抽取，自然语言处理和计算机视觉中。