This study investigates the impact of Sobolev Training on operator learning
frameworks for improving model performance. Our research reveals that
integrating derivative information into the loss function enhances the training
process, and we propose a novel framework to approximate derivatives on
irregular meshes in operator learning. Our findings are supported by both
experimental evidence and theoretical analysis. This demonstrates the
effectiveness of Sobolev Training in approximating the solution operators
between infinite-dimensional spaces.

本研究通过探讨 Sobolev 训练在操作符学习框架中对模型性能的影响，发现将导数信息集成到损失函数中能够增强训练过程，并提出了一种新的框架以在不规则网格上近似导数。实验证据和理论分析均支持了我们的研究结果，表明 Sobolev 训练在近似无穷维空间之间的解算符上是有效的。

Sobolev 训练用于运算学习

Sobolev Training for Operator Learning

An exciting branch of machine learning research focuses on methods for
learning, optimizing, and integrating unknown functions that are difficult or
costly to evaluate. A popular Bayesian approach to this problem uses a Gaussian
process (GP) to construct a posterior distribution over the function of
interest given a set of observed measurements, and selects new points to
evaluate using the statistics of this posterior. Here we extend these methods
to exploit derivative information from the unknown function. We describe
methods for Bayesian optimization (BO) and Bayesian quadrature (BQ) in settings
where first and second derivatives may be evaluated along with the function
itself. We perform sampling-based inference in order to incorporate uncertainty
over hyperparameters, and show that both hyperparameter and function
uncertainty decrease much more rapidly when using derivative information.
Moreover, we introduce techniques for overcoming ill-conditioning issues that
have plagued earlier methods for gradient-enhanced Gaussian processes and
kriging. We illustrate the efficacy of these methods using applications to real
and simulated Bayesian optimization and quadrature problems, and show that
exploting derivatives can provide substantial gains over standard methods.

本文探讨了机器学习中一种使用高斯过程对困难或昂贵评价的未知函数进行优化和集成的贝叶斯方法，并将其扩展以利用来自未知函数的导数信息，通过采样推断来融合超参数的不确定性，并且介绍了克服之前梯度增强高斯过程的 singularity 问题的技术。结果表明，利用导数信息可以在贝叶斯优化和数值积分问题中提供显着的优化性能。

利用梯度和海森矩阵在贝叶斯优化和贝叶斯积分中的应用

Exploiting gradients and Hessians in Bayesian optimization and Bayesian  quadrature

Bayesian optimization has been successful at global optimization of
expensive-to-evaluate multimodal objective functions. However, unlike most
optimization methods, Bayesian optimization typically does not use derivative
information. In this paper we show how Bayesian optimization can exploit
derivative information to decrease the number of objective function evaluations
required for good performance. In particular, we develop a novel Bayesian
optimization algorithm, the derivative-enabled knowledge-gradient (dKG), for
which we show one-step Bayes-optimality, asymptotic consistency, and greater
one-step value of information than is possible in the derivative-free setting.
Our procedure accommodates noisy and incomplete derivative information, comes
in both sequential and batch forms, and can optionally reduce the computational
cost of inference through automatically selected retention of a single
directional derivative. We also compute the d-KG acquisition function and its
gradient using a novel fast discretization-free technique. We show d-KG
provides state-of-the-art performance compared to a wide range of optimization
procedures with and without gradients, on benchmarks including logistic
regression, deep learning, kernel learning, and k-nearest neighbors.

本文提出了一种新的贝叶斯优化算法，它利用导数信息来降低目标函数评估的数量，可以适用于包括逐步和批量形式在内的嘈杂和不完整的导数信息，并且可以自动选择保留方向导数以减少推论的计算成本，该算法在众多优化过程中表现出最新的状态性能。