Gated recurrent units (GRUs) are specialized memory elements for building
recurrent neural networks. Despite their incredible success on various tasks,
including extracting dynamics underlying neural data, little is understood
about the specific dynamics representable in a GRU network. As a result, it is
both difficult to know a priori how successful a GRU network will perform on a
given task, and also their capacity to mimic the underlying behavior of their
biological counterparts. Using a continuous time analysis, we gain intuition on
the inner workings of GRU networks. We restrict our presentation to low
dimensions, allowing for a comprehensive visualization. We found a surprisingly
rich repertoire of dynamical features that includes stable limit cycles
(nonlinear oscillations), multi-stable dynamics with various topologies, and
homoclinic bifurcations. At the same time we were unable to train GRU networks
to produce continuous attractors, which are hypothesized to exist in biological
neural networks. We contextualize the usefulness of different kinds of observed
dynamics and support our claims experimentally.

使用连续时间分析，我们对 Gated recurrent units (GRUs) 的内部运作获得了直观的理解。我们发现了一些意想不到的动态特征，同时我们无法训练 GRU 网络产生连续的吸引子，这也是生物神经网络存在的假设。