We show that deep networks can be trained using Hebbian updates yielding
similar performance to ordinary back-propagation on challenging image datasets.
To overcome the unrealistic symmetry in connections between layers, implicit in
back-propagation, the feedback weights are separate from the feedforward
weights. The feedback weights are also updated with a local rule, the same as
the feedforward weights - a weight is updated solely based on the product of
activity of the units it connects. With fixed feedback weights as proposed in
Lillicrap et. al (2016) performance degrades quickly as the depth of the
network increases. If the feedforward and feedback weights are initialized with
the same values, as proposed in Zipser and Rumelhart (1990), they remain the
same throughout training thus precisely implementing back-propagation. We show
that even when the weights are initialized differently and at random, and the
algorithm is no longer performing back-propagation, performance is comparable
on challenging datasets. We also propose a cost function whose derivative can
be represented as a local Hebbian update on the last layer. Convolutional
layers are updated with tied weights across space, which is not biologically
plausible. We show that similar performance is achieved with untied layers,
also known as locally connected layers, corresponding to the connectivity
implied by the convolutional layers, but where weights are untied and updated
separately. In the linear case we show theoretically that the convergence of
the error to zero is accelerated by the update of the feedback weights.

本论文提出一种使用 Hebbian 更新训练深度网络的方法，反馈权重与前馈权重分离以克服反向传播中不真实的对称性，即使在权重不同的情况下，其性能也与常规反向传播相当，同时提出了一种可以表示为最后一层局部 Hebbian 更新的成本函数。

具有不对称连接和 Hebbian 更新 的深度学习

Deep learning with asymmetric connections and Hebbian updates

Gradient backpropagation (BP) requires symmetric feedforward and feedback
connections -- the same weights must be used for forward and backward passes.
This "weight transport problem" (Grossberg 1987) is thought to be one of the
main reasons to doubt BP's biologically plausibility. Using 15 different
classification datasets, we systematically investigate to what extent BP really
depends on weight symmetry. In a study that turned out to be surprisingly
similar in spirit to Lillicrap et al.'s demonstration (Lillicrap et al. 2014)
but orthogonal in its results, our experiments indicate that: (1) the
magnitudes of feedback weights do not matter to performance (2) the signs of
feedback weights do matter -- the more concordant signs between feedforward and
their corresponding feedback connections, the better (3) with feedback weights
having random magnitudes and 100% concordant signs, we were able to achieve the
same or even better performance than SGD. (4) some
normalizations/stabilizations are indispensable for such asymmetric BP to work,
namely Batch Normalization (BN) (Ioffe and Szegedy 2015) and/or a "Batch
Manhattan" (BM) update rule.

通过 15 个不同的分类数据集系统性地调查 BP 算法的对称性依赖程度，研究结果表明：1. 反馈权重的大小对性能没有影响；2. 反馈权重的符号对性能影响显著，符号越一致性能越好；3. 使用随机大小的反馈权重且符号完全一致时，性能相当甚至优于 SGD；4. 需要一些规范 / 稳定化才能使这种不对称 BP 算法起作用，即批量归一化（BN）和 / 或一种 “批量曼哈顿”（BM）更新规则。