In this work, we describe a novel approach to building a neural PDE solver leveraging recent advances in transformer based neural network architectures. Our model can provide solutions for different values of PDE parameters without any need for retraining the network. The training is carried out in a self-supervised manner, similar to pretraining approaches applied in language and vision tasks. We hypothesize that the model is in effect learning a family of operators (for multiple parameters) mapping the initial condition to the solution of the PDE at any future time step t. We compare this approach with the Fourier Neural Operator (FNO), and demonstrate that it can generalize over the space of PDE parameters, despite having a higher prediction error for individual parameter values compared to the FNO. We show that performance on a specific parameter can be improved by finetuning the model with very small amounts of data. We also demonstrate that the model scales with data as well as model size.

基于变压器的神经网络架构构建了一种新颖的神经PDE求解器，通过自监督学习的方式，训练了一个可以为不同的PDE参数提供解决方案的模型。与Fourier神经运算器（FNO）相比较，我们证明了这种方法可以推广到PDE参数空间，尽管对于单个参数值而言预测误差更高。我们通过使用极少量的数据微调模型来提高对特定参数的性能，同时证明了该模型的数据比例和模型规模之间的可扩展性。

自监督预训练的偏微分方程