Discovering a suitable neural network architecture for modeling complex
dynamical systems poses a formidable challenge, often involving extensive trial
and error and navigation through a high-dimensional hyper-parameter space. In
this paper, we discuss a systematic approach to constructing neural
architectures for modeling a subclass of dynamical systems, namely, Linear
Time-Invariant (LTI) systems. We use a variant of continuous-time neural
networks in which the output of each neuron evolves continuously as a solution
of a first-order or second-order Ordinary Differential Equation (ODE). Instead
of deriving the network architecture and parameters from data, we propose a
gradient-free algorithm to compute sparse architecture and network parameters
directly from the given LTI system, leveraging its properties. We bring forth a
novel neural architecture paradigm featuring horizontal hidden layers and
provide insights into why employing conventional neural architectures with
vertical hidden layers may not be favorable. We also provide an upper bound on
the numerical errors of our neural networks. Finally, we demonstrate the high
accuracy of our constructed networks on three numerical examples.

本文讨论了一种构建神经网络架构的系统方法，用于模拟一类动力系统，即线性时不变 (LTI) 系统。我们使用一种变种的连续时间神经网络，其中每个神经元的输出连续演化为一阶或二阶常微分方程的解。我们提出了一种无梯度算法，从给定的 LTI 系统中直接计算稀疏架构和网络参数，利用其特性。我们提出了一种横向隐藏层的新型神经网络架构范例，并解释了为什么使用传统的纵向隐藏层神经网络可能不是理想选择。我们还给出了该神经网络数值误差的上界，并展示了我们构建的网络在三个数值示例上的高精度。

线性动态系统的连续时间神经网络的系统构建

Systematic construction of continuous-time neural networks for linear  dynamical systems

Neural ordinary differential equations (NODEs) presented a new paradigm to
construct (continuous-time) neural networks. While showing several good
characteristics in terms of the number of parameters and the flexibility in
constructing neural networks, they also have a couple of well-known
limitations: i) theoretically NODEs learn homeomorphic mapping functions only,
and ii) sometimes NODEs show numerical instability in solving integral
problems. To handle this, many enhancements have been proposed. To our
knowledge, however, integrating attention into NODEs has been overlooked for a
while. To this end, we present a novel method of attentive dual co-evolving
NODE (ACE-NODE): one main NODE for a downstream machine learning task and the
other for providing attention to the main NODE. Our ACE-NODE supports both
pairwise and elementwise attention. In our experiments, our method outperforms
existing NODE-based and non-NODE-based baselines in almost all cases by
non-trivial margins.

本研究提出了一种新的方法：关注对偶共同进化节点 (ACE-NODE)，它是由两个节点构成的，其中一个为下游的机器学习任务提供注意力，另一个为主要节点提供注意力，支持成对注意力和元素注意力。在实验中，我们的方法在几乎所有情况下均以非微不足道的优势优于现有的基于 NODE 和非 NODE 的基线。

ACE-NODE: 关注协同演化神经常微分方程

ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations

We propose a new class of parameterizations for spatio-temporal point
processes which leverage Neural ODEs as a computational method and enable
flexible, high-fidelity models of discrete events that are localized in
continuous time and space. Central to our approach is a combination of
continuous-time neural networks with two novel neural architectures, i.e., Jump
and Attentive Continuous-time Normalizing Flows. This approach allows us to
learn complex distributions for both the spatial and temporal domain and to
condition non-trivially on the observed event history. We validate our models
on data sets from a wide variety of contexts such as seismology, epidemiology,
urban mobility, and neuroscience.

本文提出利用神经常微分方程作为计算方法的新型参数化方法来处理时空点过程，使得离散事件能够在连续时间和空间上进行灵活且精确的建模，本方法包括连续时间神经网络、Jump 以及 Attentive Continuous-time Normalizing Flows 等两种新型神经网络结构，能够处理在时间和空间领域内的复杂分布，并能够非常巧妙地对事件历史进行建模，我们在地震学、流行病学、城市移动性以及神经科学等领域的数据集上验证了我们的模型。