Neuromorphic computing and, in particular, spiking neural networks (SNNs)
have become an attractive alternative to deep neural networks for a broad range
of signal processing applications, processing static and/or temporal inputs
from different sensory modalities, including audio and vision sensors. In this
paper, we start with a description of recent advances in algorithmic and
optimization innovations to efficiently train and scale low-latency, and
energy-efficient spiking neural networks (SNNs) for complex machine learning
applications. We then discuss the recent efforts in algorithm-architecture
co-design that explores the inherent trade-offs between achieving high
energy-efficiency and low latency while still providing high accuracy and
trustworthiness. We then describe the underlying hardware that has been
developed to leverage such algorithmic innovations in an efficient way. In
particular, we describe a hybrid method to integrate significant portions of
the model's computation within both memory components as well as the sensor
itself. Finally, we discuss the potential path forward for research in building
deployable SNN systems identifying key challenges in the
algorithm-hardware-application co-design space with an emphasis on
trustworthiness.

神经形态计算及尤其是脉冲神经网络 (SNNs) 作为深度神经网络的一种有吸引力的替代方法，已广泛应用于处理来自不同感觉模态的静态和 / 或时间输入的信号处理应用。本文从算法和优化创新的最新进展开始，旨在高效训练和扩展低延迟、能源高效的复杂机器学习应用中的脉冲神经网络 (SNNs)。然后，我们讨论了算法 - 架构共设计方面的最新研究成果，探索在实现高能效和低延迟的同时仍提供高准确性和可信性之间的固有权衡。然后，我们描述了为有效利用这种算法创新而开发的基础硬件。尤其是，我们描述了一种混合方法，即在内存组件和传感器本身中整合模型计算的重要部分。最后，我们讨论了构建可部署 SNN 系统的研究前景，重点是算法 - 硬件 - 应用共设计领域的关键挑战，强调了可信性。

可扩展、能效高和值得信赖的脉冲神经网络的最新进展：从算法到技术

Recent Advances in Scalable Energy-Efficient and Trustworthy Spiking  Neural networks: from Algorithms to Technology

The attention mechanism is becoming increasingly popular in Natural Language
Processing (NLP) applications, showing superior performance than convolutional
and recurrent architectures. However, general-purpose platforms such as CPUs
and GPUs are inefficient when performing attention inference due to complicated
data movement and low arithmetic intensity. Moreover, existing NN accelerators
mainly focus on optimizing convolutional or recurrent models, and cannot
efficiently support attention. In this paper, we present SpAtten, an efficient
algorithm-architecture co-design that leverages token sparsity, head sparsity,
and quantization opportunities to reduce the attention computation and memory
access. Inspired by the high redundancy of human languages, we propose the
novel cascade token pruning to prune away unimportant tokens in the sentence.
We also propose cascade head pruning to remove unessential heads. Cascade
pruning is fundamentally different from weight pruning since there is no
trainable weight in the attention mechanism, and the pruned tokens and heads
are selected on the fly. To efficiently support them on hardware, we design a
novel top-k engine to rank token and head importance scores with high
throughput. Furthermore, we propose progressive quantization that first fetches
MSBs only and performs the computation; if the confidence is low, it fetches
LSBs and recomputes the attention outputs, trading computation for memory
reduction.
Extensive experiments on 30 benchmarks show that, on average, SpAtten reduces
DRAM access by 10.0x with no accuracy loss, and achieves 1.6x, 3.0x, 162x, 347x
speedup, and 1,4x, 3.2x, 1193x, 4059x energy savings over A3 accelerator,
MNNFast accelerator, TITAN Xp GPU, Xeon CPU, respectively.

提出了 SpAtten，这是一种高效的算法 - 架构协同设计，利用标记稀疏性、头部稀疏性和量化机会，通过级联剪枝和渐进式量化来减少注意力计算和内存访问，实验结果证明其在 30 个基准测试中平均减少 DRAM 访问 10 倍，并且相对于其他加速器和处理器能达到巨大的加速和能耗降低。