Scaling hyperparameter optimisation to very large datasets remains an open
problem in the Gaussian process community. This paper focuses on iterative
methods, which use linear system solvers, like conjugate gradients, alternating
projections or stochastic gradient descent, to construct an estimate of the
marginal likelihood gradient. We discuss three key improvements which are
applicable across solvers: (i) a pathwise gradient estimator, which reduces the
required number of solver iterations and amortises the computational cost of
making predictions, (ii) warm starting linear system solvers with the solution
from the previous step, which leads to faster solver convergence at the cost of
negligible bias, (iii) early stopping linear system solvers after a limited
computational budget, which synergises with warm starting, allowing solver
progress to accumulate over multiple marginal likelihood steps. These
techniques provide speed-ups of up to $72\times$ when solving to tolerance, and
decrease the average residual norm by up to $7\times$ when stopping early.

将超参数优化扩展到非常大的数据集仍然是高斯过程领域中一个未解决的问题。本文侧重于使用线性系统求解器（如共轭梯度、交替投影或随机梯度下降）构建边缘似然梯度的估计的迭代方法。我们讨论了三个可用于所有求解器的关键改进：（i）路径梯度估计器，减少了求解器迭代次数的要求并分摊了预测的计算成本，（ii）使用前一步的解来热启动线性系统求解器，导致更快的求解器收敛速度且误差可以忽略，（iii）在有限的计算预算后及早停止线性系统求解器，与热启动相协同，允许求解器进展在多个边缘似然步骤中积累。这些技术在达到容忍度时提供了高达 72 倍的加速，并在早停止时将平均残差范数降低了 7 倍。

改进迭代高斯过程中的超参数优化的线性系统求解器

Improving Linear System Solvers for Hyperparameter Optimisation in  Iterative Gaussian Processes

The partially observable generalized linear model (POGLM) is a powerful tool
for understanding neural connectivity under the assumption of existing hidden
neurons. With spike trains only recorded from visible neurons, existing works
use variational inference to learn POGLM meanwhile presenting the difficulty of
learning this latent variable model. There are two main issues: (1) the sampled
Poisson hidden spike count hinders the use of the pathwise gradient estimator
in VI; and (2) the existing design of the variational model is neither
expressive nor time-efficient, which further affects the performance. For (1),
we propose a new differentiable POGLM, which enables the pathwise gradient
estimator, better than the score function gradient estimator used in existing
works. For (2), we propose the forward-backward message-passing sampling scheme
for the variational model. Comprehensive experiments show that our
differentiable POGLMs with our forward-backward message passing produce a
better performance on one synthetic and two real-world datasets. Furthermore,
our new method yields more interpretable parameters, underscoring its
significance in neuroscience.

部分可观察的广义线性模型（POGLM）是一个强大的工具，用于理解存在隐藏神经元的神经连接性。本文提出了一种新的可微分 POGLM，使得路径梯度估计器比现有方法中使用的得分函数梯度估计器更好。同时，我们还提出了一种基于前向 - 后向消息传递采样方案的变分模型，利用全面的实验验证了我们的方法在合成数据集和真实数据集上更好的性能，并且产生更可解释的参数，凸显其在神经科学中的重要性。