This work contributes an event-driven visual-tactile perception system,
comprising a novel biologically-inspired tactile sensor and multi-modal
spike-based learning. Our neuromorphic fingertip tactile sensor, NeuTouch,
scales well with the number of taxels thanks to its event-based nature.
Likewise, our Visual-Tactile Spiking Neural Network (VT-SNN) enables fast
perception when coupled with event sensors. We evaluate our visual-tactile
system (using the NeuTouch and Prophesee event camera) on two robot tasks:
container classification and rotational slip detection. On both tasks, we
observe good accuracies relative to standard deep learning methods. We have
made our visual-tactile datasets freely-available to encourage research on
multi-modal event-driven robot perception, which we believe is a promising
approach towards intelligent power-efficient robot systems.

本文介绍了一个基于事件触觉感测器和多模式脉冲神经网络的事件驱动的视觉触觉感知系统，评估了其在两个机器人任务上的表现，并提供了可用于其他研究的数据集。

机器人事件驱动的视觉和触觉感知与学习

Event-Driven Visual-Tactile Sensing and Learning for Robots

The integration of visual-tactile stimulus is common while humans performing
daily tasks. In contrast, using unimodal visual or tactile perception limits
the perceivable dimensionality of a subject. However, it remains a challenge to
integrate the visual and tactile perception to facilitate robotic tasks. In
this paper, we propose a novel framework for the cross-modal sensory data
generation for visual and tactile perception. Taking texture perception as an
example, we apply conditional generative adversarial networks to generate
pseudo visual images or tactile outputs from data of the other modality.
Extensive experiments on the ViTac dataset of cloth textures show that the
proposed method can produce realistic outputs from other sensory inputs. We
adopt the structural similarity index to evaluate similarity of the generated
output and real data and results show that realistic data have been generated.
Classification evaluation has also been performed to show that the inclusion of
generated data can improve the perception performance. The proposed framework
has potential to expand datasets for classification tasks, generate sensory
outputs that are not easy to access, and also advance integrated visual-tactile
perception.

本文提出了一种新的框架，利用条件生成对抗网络生成视觉或触觉图像，以实现视觉和触觉知觉的跨模态感知，并在 ViTac 数据集上进行了广泛实验，结果表明该方法能够生成逼真的数据并且有潜力扩大分类任务的数据集，生成不易获取的感官输出，并推进视觉 - 触觉知觉的整合。