As intelligent systems become increasingly important in our daily lives, new
ways of interaction are needed. Classical user interfaces pose issues for the
physically impaired and are partially not practical or convenient. Gesture
recognition is an alternative, but often not reactive enough when conventional
cameras are used. This work proposes a Spiking Convolutional Neural Network,
processing event- and depth data for gesture recognition. The network is
simulated using the open-source neuromorphic computing framework LAVA for
offline training and evaluation on an embedded system. For the evaluation three
open source data sets are used. Since these do not represent the applied
bi-modality, a new data set with synchronized event- and depth data was
recorded. The results show the viability of temporal encoding on depth
information and modality fusion, even on differently encoded data, to be
beneficial to network performance and generalization capabilities.

智能系统在我们日常生活中变得越来越重要，需要新的交互方式。本文提出的脉冲卷积神经网络利用事件和深度数据进行手势识别，通过在嵌入式系统上进行离线训练和评估，并使用开源的神经形态计算框架 LAVA 进行模拟。研究结果表明，在深度信息和模态融合中，时间编码及不同编码的数据对网络性能和泛化能力有积极效果。