Gaussian processes are a versatile probabilistic machine learning model whose
effectiveness often depends on good hyperparameters, which are typically
learned by maximising the marginal likelihood. In this work, we consider
iterative methods, which use iterative linear system solvers to approximate
marginal likelihood gradients up to a specified numerical precision, allowing a
trade-off between compute time and accuracy of a solution. We introduce a
three-level hierarchy of marginal likelihood optimisation for iterative
Gaussian processes, and identify that the computational costs are dominated by
solving sequential batches of large positive-definite systems of linear
equations. We then propose to amortise computations by reusing solutions of
linear system solvers as initialisations in the next step, providing a
$\textit{warm start}$. Finally, we discuss the necessary conditions and
quantify the consequences of warm starts and demonstrate their effectiveness on
regression tasks, where warm starts achieve the same results as the
conventional procedure while providing up to a $16 \times$ average speed-up
among datasets.

通过引入多级边际似然优化和温启动，我们提出了一种迭代高斯过程模型，可以通过重复使用线性系统求解器的解作为下一步的初始化值，在计算不变性的情况下有效地提高回归任务的计算速度。

迭代高斯过程的热启动边缘似然优化

Warm Start Marginal Likelihood Optimisation for Iterative Gaussian  Processes

The barren plateau phenomenon, characterized by loss gradients that vanish
exponentially with system size, poses a challenge to scaling variational
quantum algorithms. Here we explore the potential of warm starts, whereby one
initializes closer to a solution in the hope of enjoying larger loss variances.
Focusing on an iterative variational method for learning shorter-depth circuits
for quantum real and imaginary time evolution we conduct a case study to
elucidate the potential and limitations of warm starts. We start by proving
that the iterative variational algorithm will exhibit substantial (at worst
vanishing polynomially in system size) gradients in a small region around the
initializations at each time-step. Convexity guarantees for these regions are
then established, suggesting trainability for polynomial size time-steps.
However, our study highlights scenarios where a good minimum shifts outside the
region with trainability guarantees. Our analysis leaves open the question
whether such minima jumps necessitate optimization across barren plateau
landscapes or whether there exist gradient flows, i.e., fertile valleys away
from the plateau with substantial gradients, that allow for training.

通过研究温起始对学习量子实时和虚时间演化中较短深度电路的影响，探索缓解荒漠高原现象的潜力和局限性。

变分量子模拟：热启动理解案例研究

Variational quantum simulation: a case study for understanding warm  starts

We introduce a machine-learning framework to warm-start fixed-point
optimization algorithms. Our architecture consists of a neural network mapping
problem parameters to warm starts, followed by a predefined number of
fixed-point iterations. We propose two loss functions designed to either
minimize the fixed-point residual or the distance to a ground truth solution.
In this way, the neural network predicts warm starts with the end-to-end goal
of minimizing the downstream loss. An important feature of our architecture is
its flexibility, in that it can predict a warm start for fixed-point algorithms
run for any number of steps, without being limited to the number of steps it
has been trained on. We provide PAC-Bayes generalization bounds on unseen data
for common classes of fixed-point operators: contractive, linearly convergent,
and averaged. Applying this framework to well-known applications in control,
statistics, and signal processing, we observe a significant reduction in the
number of iterations and solution time required to solve these problems,
through learned warm starts.

我们介绍了一个机器学习框架，用于启动固定点优化算法；我们的架构由一个神经网络和一定数量的固定点迭代组成；我们提出了两个损失函数，其中一个用于最小化固定点残差，另一个用于最小化到参考解的距离；我们的架构具有灵活性，可以预测任意步骤下的固定点算法的启动；我们提供了对常见类别的固定点算子（收敛、线性和平均）的未见数据的 PAC-Bayes 泛化界限；将该框架应用于控制、统计和信号处理等领域的知名应用可以显著减少求解这些问题所需的迭代次数和解决时间。