With the help of special neuromorphic hardware, spiking neural networks
(SNNs) are expected to realize artificial intelligence with less energy
consumption. It provides a promising energy-efficient way for realistic control
tasks by combing SNNs and deep reinforcement learning (RL). There are only a
few existing SNN-based RL methods at present. Most of them either lack
generalization ability or employ Artificial Neural Networks (ANNs) to estimate
value function in training. The former needs to tune numerous hyper-parameters
for each scenario, and the latter limits the application of different types of
RL algorithm and ignores the large energy consumption in training. To develop a
robust spike-based RL method, we draw inspiration from non-spiking interneurons
found in insects and propose the deep spiking Q-network (DSQN), using the
membrane voltage of non-spiking neurons as the representation of Q-value, which
can directly learn robust policies from high-dimensional sensory inputs using
end-to-end RL. Experiments conducted on 17 Atari games demonstrate the
effectiveness of DSQN by outperforming the ANN-based deep Q-network (DQN) in
most games. Moreover, the experimental results show superior learning stability
and robustness to adversarial attacks of DSQN.

本研究提出了一种名为 DSQN 的深度尖峰 Q 网络，使用非尖峰神经元的膜电压作为 Q 值的表示，可以从高维度的感官输入中直接学习稳健的决策，并在 17 个 Atari 游戏中表现出优异的性能，具有更好的学习稳定性和对抗攻击鲁棒性。