Predictive coding (PC) accounts of perception now form one of the dominant
computational theories of the brain, where they prescribe a general algorithm
for inference and learning over hierarchical latent probabilistic models.
Despite this, they have enjoyed little export to the broader field of machine
learning, where comparative generative modelling techniques have flourished. In
part, this has been due to the poor performance of models trained with PC when
evaluated by both sample quality and marginal likelihood. By adopting the
perspective of PC as a variational Bayes algorithm under the Laplace
approximation, we identify the source of these deficits to lie in the exclusion
of an associated Hessian term in the PC objective function, which would
otherwise regularise the sharpness of the probability landscape and prevent
over-certainty in the approximate posterior. To remedy this, we make three
primary contributions: we begin by suggesting a simple Monte Carlo estimated
evidence lower bound which relies on sampling from the Hessian-parameterised
variational posterior. We then derive a novel block diagonal approximation to
the full Hessian matrix that has lower memory requirements and favourable
mathematical properties. Lastly, we present an algorithm that combines our
method with standard PC to reduce memory complexity further. We evaluate models
trained with our approach against the standard PC framework on image benchmark
datasets. Our approach produces higher log-likelihoods and qualitatively better
samples that more closely capture the diversity of the data-generating
distribution.

通过将预测编码视为拉普拉斯近似下的变分贝叶斯算法，我们发现其目标函数中排除与之关联的 Hessian 项是其性能不佳的根源。借此，我们提出了三个主要贡献：提出了一个简单的蒙特卡罗估计证据下界，该证据下界依赖于从 Hessian 参数化的变分后验中进行采样；推导出了全 Hessian 矩阵的新型块对角线逼近；介绍了一种将我们的方法与标准预测编码相结合以进一步降低内存复杂度的算法。我们在图像基准数据集上评估了我们的方法训练的模型，并将其与标准预测编码框架进行了对比。我们的方法产生了更高的对数似然值和更好的样本，更好地捕获了数据生成分布的多样性。