We propose a neuroscience-inspired Solo Pass Embedded Learning Algorithm
(SPELA). SPELA is a prime candidate for training and inference applications in
Edge AI devices. At the same time, SPELA can optimally cater to the need for a
framework to study perceptual representation learning and formation. SPELA has
distinctive features such as neural priors (in the form of embedded vectors),
no weight transport, no update locking of weights, complete local Hebbian
learning, single forward pass with no storage of activations, and single weight
update per sample. Juxtaposed with traditional approaches, SPELA operates
without the need for backpropagation. We show that our algorithm can perform
nonlinear classification on a noisy boolean operation dataset. Additionally, we
exhibit high performance using SPELA across MNIST, KMNIST, and Fashion MNIST.
Lastly, we show the few-shot and 1-epoch learning capabilities of SPELA on
MNIST, KMNIST, and Fashion MNIST, where it consistently outperforms
backpropagation.

我们提出了一种以神经科学为灵感的单次传递嵌入式学习算法（SPELA），它是培训和推理应用于边缘 AI 设备的优选候选者，同时 SPELA 可以最佳地满足对感知表示学习和形成的研究框架的需求。与传统方法相比，SPELA 无需反向传播，我们展示了该算法在噪声布尔操作数据集上能够进行非线性分类的性能，并且在 MNIST、KMNIST 和 Fashion MNIST 上表现出高性能，最后，我们展示了 SPELA 在 MNIST、KMNIST 和 Fashion MNIST 上的几个 shot 和 1 epoch 学习能力，其中它始终优于反向传播。

单次前向传播的表示学习

Representation Learning Using a Single Forward Pass

Model architectures such as wav2vec 2.0 and HuBERT have been proposed to
learn speech representations from audio waveforms in a self-supervised manner.
When these models are combined with downstream tasks such as speech
recognition, they have been shown to provide state-of-the-art performance.
However, these models use a large number of parameters, the smallest version of
which has about 95 million parameters. This constitutes a challenge for edge AI
device deployments. In this paper, we use knowledge distillation to reduce the
original model size by about 75% while maintaining similar performance levels.
Moreover, we use wav2vec 2.0 and HuBERT models for distillation and present a
comprehensive performance analysis through our experiments where we fine-tune
the distilled models on single task and multi-task frameworks separately. In
particular, our experiments show that fine-tuning the distilled models on
keyword spotting and speaker verification tasks result in only 0.1% accuracy
and 0.9% equal error rate degradations, respectively.

本论文中，我们使用知识蒸馏通过压缩 wav2vec 2.0 和 HuBERT 语音模型约 75% 的参数量，并分别在单任务和多任务框架上进行微调实验来进行全面的性能分析，实验结果表明，在关键词检测和说话人验证任务中，微调经过知识蒸馏的模型只有 0.1% 的准确率和 0.9% 的等误差率下降。