The recent successes in analyzing images with deep neural networks are almost
exclusively achieved with Convolutional Neural Networks (CNNs). The training of
these CNNs, and in fact of all deep neural network architectures, uses the
backpropagation algorithm where the output of the network is compared with the
desired result and the difference is then used to tune the weights of the
network towards the desired outcome. In a 2022 preprint, Geoffrey Hinton
suggested an alternative way of training which passes the desired results
together with the images at the input of the network. This so called Forward
Forward (FF) algorithm has up to now only been used in fully connected
networks. In this paper, we show how the FF paradigm can be extended to CNNs.
Our FF-trained CNN, featuring a novel spatially-extended labeling technique,
achieves a classification accuracy of 99.0% on the MNIST hand-written digits
dataset. We show how different hyperparameters affect the performance of the
proposed algorithm and compare the results with CNN trained with the standard
backpropagation approach. Furthermore, we use Class Activation Maps to
investigate which type of features are learnt by the FF algorithm.

探索了前馈算法在卷积神经网络中的应用，并使用新的标记技术，在 MNIST 手写数字数据集上实现了 99％的分类准确率，并比较了不同超参数对算法性能的影响。

使用前向 - 前向算法训练卷积神经网络

Training Convolutional Neural Networks with the Forward-Forward  algorithm

The recently proposed multi-layer sparse model has raised insightful
connections between sparse representations and convolutional neural networks
(CNN). In its original conception, this model was restricted to a cascade of
convolutional synthesis representations. In this paper, we start by addressing
a more general model, revealing interesting ties to fully connected networks.
We then show that this multi-layer construction admits a brand new
interpretation in a unique symbiosis between synthesis and analysis models:
while the deepest layer indeed provides a synthesis representation, the
mid-layers decompositions provide an analysis counterpart. This new perspective
exposes the suboptimality of previously proposed pursuit approaches, as they do
not fully leverage all the information comprised in the model constraints.
Armed with this understanding, we address fundamental theoretical issues,
revisiting previous analysis and expanding it. Motivated by the limitations of
previous algorithms, we then propose an integrated - holistic - alternative
that estimates all representations in the model simultaneously, and analyze all
these different schemes under stochastic noise assumptions. Inspired by the
synthesis-analysis duality, we further present a Holistic Pursuit algorithm,
which alternates between synthesis and analysis sparse coding steps, eventually
solving for the entire model as a whole, with provable improved performance.
Finally, we present numerical results that demonstrate the practical advantages
of our approach.

本文研究了多层稀疏模型的构建与应用，将其与卷积神经网络和全连接网络进行了关联，提出一种新的综合算法 ——Holistic Pursuit 算法，可以同时估计所有模型中的表示，并在实验中取得了良好的表现。

多层稀疏编码：全面的方法

Multi-Layer Sparse Coding: The Holistic Way

We propose ways to improve the performance of fully connected networks. We
found that two approaches in particular have a strong effect on performance:
linear bottleneck layers and unsupervised pre-training using autoencoders
without hidden unit biases. We show how both approaches can be related to
improving gradient flow and reducing sparsity in the network. We show that a
fully connected network can yield approximately 70% classification accuracy on
the permutation-invariant CIFAR-10 task, which is much higher than the current
state-of-the-art. By adding deformations to the training data, the fully
connected network achieves 78% accuracy, which is just 10% short of a decent
convolutional network.

探讨改进全连接网络的性能的方法，提出了线性瓶颈层和非监督预训练方法的影响，探索它们如何通过提高梯度流和减少稀疏性来改进网络，展示全连接网络在置换不变的 CIFAR-10 任务上达到了约 70% 的分类准确率，添加变形数据后达到了 78% 的准确率，接近卷积网络的水平。