In this work, we tackle the problems of efficiency and scalability for
predictive coding networks in machine learning. To do so, we first propose a
library called PCX, whose focus lies on performance and simplicity, and
provides a user-friendly, deep-learning oriented interface. Second, we use PCX
to implement a large set of benchmarks for the community to use for their
experiments. As most works propose their own tasks and architectures, do not
compare one against each other, and focus on small-scale tasks, a simple and
fast open-source library adopted by the whole community would address all of
these concerns. Third, we perform extensive benchmarks using multiple
algorithms, setting new state-of-the-art results in multiple tasks and
datasets, as well as highlighting limitations inherent to PC that should be
addressed. Thanks to the efficiency of PCX, we are able to analyze larger
architectures than commonly used, providing baselines to galvanize community
efforts towards one of the main open problems in the field: scalability. The
code for PCX is available at \textit{this https URL}.

在这项研究中，我们提出了一个名为 PCX 的库，旨在解决机器学习中预测编码网络的效率和可扩展性问题，并使用 PCX 实现了一系列用于实验的基准测试。我们通过广泛的基准测试和算法比较，展示了 PCX 的高效性，并指出了需要解决的潜在局限性，为解决该领域的主要开放问题之一 —— 可扩展性提供了基准。

预测编码网络基准测试 —— 简易版

Benchmarking Predictive Coding Networks -- Made Simple

Predictive Coding Networks (PCNs) aim to learn a generative model of the
world. Given observations, this generative model can then be inverted to infer
the causes of those observations. However, when training PCNs, a noticeable
pathology is often observed where inference accuracy peaks and then declines
with further training. This cannot be explained by overfitting since both
training and test accuracy decrease simultaneously. Here we provide a thorough
investigation of this phenomenon and show that it is caused by an imbalance
between the speeds at which the various layers of the PCN converge. We
demonstrate that this can be prevented by regularising the weight matrices at
each layer: by restricting the relative size of matrix singular values, we
allow the weight matrix to change but restrict the overall impact which a layer
can have on its neighbours. We also demonstrate that a similar effect can be
achieved through a more biologically plausible and simple scheme of just
capping the weights.

本文研究了 Predictive Coding Networks 训练时出现的中途准确度急剧下降现象，发现它是由于每一层收敛速度出现不平衡所导致的，通过正则化每层的权重矩阵或者限制权重范围可以解决这一问题。

预测编码网络中分类准确度的预防措施

Preventing Deterioration of Classification Accuracy in Predictive Coding Networks

The quality of data representation in deep learning methods is directly
related to the prior model imposed on the representations; however, generally
used fixed priors are not capable of adjusting to the context in the data. To
address this issue, we propose deep predictive coding networks, a hierarchical
generative model that empirically alters priors on the latent representations
in a dynamic and context-sensitive manner. This model captures the temporal
dependencies in time-varying signals and uses top-down information to modulate
the representation in lower layers. The centerpiece of our model is a novel
procedure to infer sparse states of a dynamic model which is used for feature
extraction. We also extend this feature extraction block to introduce a pooling
function that captures locally invariant representations. When applied on a
natural video data, we show that our method is able to learn high-level visual
features. We also demonstrate the role of the top-down connections by showing
the robustness of the proposed model to structured noise.

提出深度预测编码网络，用于学习具有动态情境感知的表征学习，实现了高效的特征提取与局部不变性表示，同时具有良好的结构噪声适应能力。