Recently, the neuromorphic vision sensor has received more and more interest.
However, the neuromorphic data consists of asynchronous event spikes, which is
not natural and difficult to construct a benchmark, thus limiting the
neuromorphic data understanding for "unseen" objects by deep learning.
Zero-shot and few-shot learning via Contrastive Vision-Language Pre-training
(CLIP) have shown inspirational performance in 2D frame image recognition. To
handle "unseen" recognition for the neuromorphic data, in this paper, we
propose NeuroCLIP, which transfers the CLIP's 2D pre-trained knowledge to event
spikes. To improve the few-shot performance, we also provide an inter-timestep
adapter based on a spiking neural network. Our code is open-sourced at
this https URL

本研究提出了一种名为 NeuroCLIP 的新方法，通过将 Contrastive Vision-Language Pre-training（CLIP）的二维预训练知识转移到事件脉冲上，以实现对神经形态数据中 “未见过” 的对象的识别，同时提供了一个基于触发神经网络的逐时间步适配器，以提高 few-shot 分类效果。

NeuroCLIP: CLIP 和 SNN 辅助下的神经形态数据理解

NeuroCLIP: Neuromorphic Data Understanding by CLIP and SNN

Neuromorphic vision sensor is a new bio-inspired imaging paradigm that
reports asynchronous, continuously per-pixel brightness changes called `events'
with high temporal resolution and high dynamic range. So far, the event-based
image reconstruction methods are based on artificial neural networks (ANN) or
hand-crafted spatiotemporal smoothing techniques. In this paper, we first
implement the image reconstruction work via fully spiking neural network (SNN)
architecture. As the bio-inspired neural networks, SNNs operating with
asynchronous binary spikes distributed over time, can potentially lead to
greater computational efficiency on event-driven hardware. We propose a novel
Event-based Video reconstruction framework based on a fully Spiking Neural
Network (EVSNN), which utilizes Leaky-Integrate-and-Fire (LIF) neuron and
Membrane Potential (MP) neuron. We find that the spiking neurons have the
potential to store useful temporal information (memory) to complete such
time-dependent tasks. Furthermore, to better utilize the temporal information,
we propose a hybrid potential-assisted framework (PA-EVSNN) using the membrane
potential of spiking neuron. The proposed neuron is referred as Adaptive
Membrane Potential (AMP) neuron, which adaptively updates the membrane
potential according to the input spikes. The experimental results demonstrate
that our models achieve comparable performance to ANN-based models on IJRR,
MVSEC, and HQF datasets. The energy consumptions of EVSNN and PA-EVSNN are
19.36$\times$ and 7.75$\times$ more computationally efficient than their ANN
architectures, respectively.

本文提出了一种基于全脉冲神经网络 (SNN) 架构的事件相关视频重建框架 (EVSNN)，并采用 Leaky-Integrate-and-Fire (LIF) 神经元和自适应膜电位中间电位助推框架 (PA-EVSNN)，通过对事件相关图像信息的处理， 实现与基于人工神经网络（ANN）的模型相当的性能，而且在事件驱动硬件上具有更高的计算效率。