We introduce a novel two-level overlapping additive Schwarz preconditioner
for accelerating the training of scientific machine learning applications. The
design of the proposed preconditioner is motivated by the nonlinear two-level
overlapping additive Schwarz preconditioner. The neural network parameters are
decomposed into groups (subdomains) with overlapping regions. In addition, the
network's feed-forward structure is indirectly imposed through a novel
subdomain-wise synchronization strategy and a coarse-level training step.
Through a series of numerical experiments, which consider physics-informed
neural networks and operator learning approaches, we demonstrate that the
proposed two-level preconditioner significantly speeds up the convergence of
the standard (LBFGS) optimizer while also yielding more accurate machine
learning models. Moreover, the devised preconditioner is designed to take
advantage of model-parallel computations, which can further reduce the training
time.

提出了一种新颖的双层重叠添加 Schwarz 预条件器，用于加速科学机器学习应用的训练，通过在网络参数中引入重叠区域的组（子域），间接通过一种新颖的子域同步策略和粗粒度训练步骤实现网络的前馈结构，并通过一系列数值实验表明，所提出的双层预条件器显著加速了标准（LBFGS）优化器的收敛速度，同时产生了更准确的机器学习模型，此外，所设计的预条件器可以充分利用模型并行计算，从而进一步减少训练时间。

用于训练科学机器学习应用的两级重叠加法 Schwarz 预条件器

Two-level overlapping additive Schwarz preconditioner for training  scientific machine learning applications

Pre-trained machine learning (ML) models have shown great performance for a
wide range of applications, in particular in natural language processing (NLP)
and computer vision (CV). Here, we study how pre-training could be used for
scientific machine learning (SciML) applications, specifically in the context
of transfer learning. We study the transfer behavior of these models as (i) the
pre-trained model size is scaled, (ii) the downstream training dataset size is
scaled, (iii) the physics parameters are systematically pushed out of
distribution, and (iv) how a single model pre-trained on a mixture of different
physics problems can be adapted to various downstream applications. We find
that-when fine-tuned appropriately-transfer learning can help reach desired
accuracy levels with orders of magnitude fewer downstream examples (across
different tasks that can even be out-of-distribution) than training from
scratch, with consistent behavior across a wide range of downstream examples.
We also find that fine-tuning these models yields more performance gains as
model size increases, compared to training from scratch on new downstream
tasks. These results hold for a broad range of PDE learning tasks. All in all,
our results demonstrate the potential of the "pre-train and fine-tune" paradigm
for SciML problems, demonstrating a path towards building SciML foundation
models. We open-source our code for reproducibility.

研究表明，预训练模型可以应用于科学机器学习任务中的迁移学习，可在细调适当的预训练模型下，以比从头开始训练少得多的下游示例实现期望的准确度水平。模型的规模越大，用 fine-tuning 方法会获得更好的性能提升，这为 SciML 问题的建模奠定了基础。