With the help of special neuromorphic hardware, spiking neural networks
(SNNs) are expected to realize artificial intelligence (AI) with less energy
consumption. It provides a promising energy-efficient way for realistic control
tasks by combining SNNs with deep reinforcement learning (DRL). In this paper,
we focus on the task where the agent needs to learn multi-dimensional
deterministic policies to control, which is very common in real scenarios.
Recently, the surrogate gradient method has been utilized for training
multi-layer SNNs, which allows SNNs to achieve comparable performance with the
corresponding deep networks in this task. Most existing spike-based RL methods
take the firing rate as the output of SNNs, and convert it to represent
continuous action space (i.e., the deterministic policy) through a
fully-connected (FC) layer. However, the decimal characteristic of the firing
rate brings the floating-point matrix operations to the FC layer, making the
whole SNN unable to deploy on the neuromorphic hardware directly. To develop a
fully spiking actor network without any floating-point matrix operations, we
draw inspiration from the non-spiking interneurons found in insects and employ
the membrane voltage of the non-spiking neurons to represent the action. Before
the non-spiking neurons, multiple population neurons are introduced to decode
different dimensions of actions. Since each population is used to decode a
dimension of action, we argue that the neurons in each population should be
connected in time domain and space domain. Hence, the intra-layer connections
are used in output populations to enhance the representation capacity. Finally,
we propose a fully spiking actor network with intra-layer connections
(ILC-SAN).

借鉴昆虫中的非脉冲神经元，我们提出了一种完全脉冲的演员网络 (ILC-SAN)，其中利用了神经形态硬件、脉冲神经网络和深度强化学习，旨在实现高效的人工智能控制任务。