Event-based cameras offer a low-power alternative to frame-based cameras for
capturing high-speed motion and high dynamic range scenes. They provide
asynchronous streams of sparse events. Spiking Neural Networks (SNNs) with
their asynchronous event-driven compute, show great potential for extracting
the spatio-temporal features from these event streams. In contrast, the
standard Analog Neural Networks (ANNs1) fail to process event data effectively.
However, training SNNs is difficult due to additional trainable parameters
(thresholds and leaks), vanishing spikes at deeper layers, non-differentiable
binary activation function etc. Moreover, an additional data structure
"membrane potential" responsible for keeping track of temporal information,
must be fetched and updated at every timestep in SNNs. To overcome these, we
propose a novel SNN-ANN hybrid architecture that combines the strengths of
both. Specifically, we leverage the asynchronous compute capabilities of SNN
layers to effectively extract the input temporal information. While the ANN
layers offer trouble-free training and implementation on standard machine
learning hardware such as GPUs. We provide extensive experimental analysis for
assigning each layer to be spiking or analog in nature, leading to a network
configuration optimized for performance and ease of training. We evaluate our
hybrid architectures for optical flow estimation using event-data on DSEC-flow
and Mutli-Vehicle Stereo Event-Camera (MVSEC) datasets. The results indicate
that our configured hybrid architectures outperform the state-of-the-art
ANN-only, SNN-only and past hybrid architectures both in terms of accuracy and
efficiency. Specifically, our hybrid architecture exhibit a 31% and 24.8% lower
average endpoint error (AEE) at 2.1x and 3.1x lower energy, compared to an
SNN-only architecture on DSEC and MVSEC datasets, respectively.

该研究提出了一种新的 SNN-ANN 混合体系结构，该体系结构结合了事件图像和异步计算的优势，用于光流估计，实验结果表明，与以往的混合体系结构，ANN-only 和 SNN-only 体系结构相比，该混合体系结构在准确性和效率方面均有显著提升。