Neural ordinary differential equations (ODEs) have attracted much attention
as continuous-time counterparts of deep residual neural networks (NNs), and
numerous extensions for recurrent NNs have been proposed. Since the 1980s, ODEs
have also been used to derive theoretical results for NN learning rules, e.g.,
the famous connection between Oja's rule and principal component analysis. Such
rules are typically expressed as additive iterative update processes which have
straightforward ODE counterparts. Here we introduce a novel combination of
learning rules and Neural ODEs to build continuous-time sequence processing
nets that learn to manipulate short-term memory in rapidly changing synaptic
connections of other nets. This yields continuous-time counterparts of Fast
Weight Programmers and linear Transformers. Our novel models outperform the
best existing Neural Controlled Differential Equation based models on various
time series classification tasks, while also addressing their fundamental
scalability limitations. Our code is public.

本研究将学习规则和神经 ODE 相结合，构建了连续时间序列处理网络，学习如何在其他网络的快速变化的突触连接中操作短期记忆，这产生了快速权重程序员和线性变压器的连续时间对应物。该模型在各种时间序列分类任务中优于现有的神经控制微分方程模型，同时也解决了它们的根本可扩展性限制。