Discovering a suitable neural network architecture for modeling complex
dynamical systems poses a formidable challenge, often involving extensive trial
and error and navigation through a high-dimensional hyper-parameter space. In
this paper, we discuss a systematic approach to constructing neural
architectures for modeling a subclass of dynamical systems, namely, Linear
Time-Invariant (LTI) systems. We use a variant of continuous-time neural
networks in which the output of each neuron evolves continuously as a solution
of a first-order or second-order Ordinary Differential Equation (ODE). Instead
of deriving the network architecture and parameters from data, we propose a
gradient-free algorithm to compute sparse architecture and network parameters
directly from the given LTI system, leveraging its properties. We bring forth a
novel neural architecture paradigm featuring horizontal hidden layers and
provide insights into why employing conventional neural architectures with
vertical hidden layers may not be favorable. We also provide an upper bound on
the numerical errors of our neural networks. Finally, we demonstrate the high
accuracy of our constructed networks on three numerical examples.

本文讨论了一种构建神经网络架构的系统方法，用于模拟一类动力系统，即线性时不变 (LTI) 系统。我们使用一种变种的连续时间神经网络，其中每个神经元的输出连续演化为一阶或二阶常微分方程的解。我们提出了一种无梯度算法，从给定的 LTI 系统中直接计算稀疏架构和网络参数，利用其特性。我们提出了一种横向隐藏层的新型神经网络架构范例，并解释了为什么使用传统的纵向隐藏层神经网络可能不是理想选择。我们还给出了该神经网络数值误差的上界，并展示了我们构建的网络在三个数值示例上的高精度。