Neuromorphic visual sensors are artificial retinas that output sequences of
asynchronous events when brightness changes occur in the scene. These sensors
offer many advantages including very high temporal resolution, no motion blur
and smart data compression ideal for real-time processing. In this study, we
introduce an event-based dataset on fine-grained manipulation actions and
perform an experimental study on the use of transformers for action prediction
with events. There is enormous interest in the fields of cognitive robotics and
human-robot interaction on understanding and predicting human actions as early
as possible. Early prediction allows anticipating complex stages for planning,
enabling effective and real-time interaction. Our Transformer network uses
events to predict manipulation actions as they occur, using online inference.
The model succeeds at predicting actions early on, building up confidence over
time and achieving state-of-the-art classification. Moreover, the
attention-based transformer architecture allows us to study the role of the
spatio-temporal patterns selected by the model. Our experiments show that the
Transformer network captures action dynamic features outperforming video-based
approaches and succeeding with scenarios where the differences between actions
lie in very subtle cues. Finally, we release the new event dataset, which is
the first in the literature for manipulation action recognition. Code will be
available at this https URL

该研究介绍了一种基于事件的数据集，用于细粒度操作动作的预测，使用的模型是基于事件进行在线推理的 Transformer 网络，该模型成功地预测了动作，并通过时间逐渐增加信心，达到了最先进的分类效果，并且优于基于视频的方法，适用于具有非常微妙差异的场景。同时，该研究还发布了第一个用于操作动作识别的事件数据集。