We introduce a new class of vision-based sensor and associated algorithmic
processes that combine visual imaging with high-resolution tactile sending, all
in a uniform hardware and computational architecture. We demonstrate the
sensor's efficacy for both multi-modal object recognition and metrology. Object
recognition is typically formulated as an unimodal task, but by combining two
sensor modalities we show that we can achieve several significant performance
improvements. This sensor, named the See-Through-your-Skin sensor (STS), is
designed to provide rich multi-modal sensing of contact surfaces. Inspired by
recent developments in optical tactile sensing technology, we address a key
missing feature of these sensors: the ability to capture a visual perspective
of the region beyond the contact surface. Whereas optical tactile sensors are
typically opaque, we present a sensor with a semitransparent skin that has the
dual capabilities of acting as a tactile sensor and/or as a visual camera
depending on its internal lighting conditions. This paper details the design of
the sensor, showcases its dual sensing capabilities, and presents a deep
learning architecture that fuses vision and touch. We validate the ability of
the sensor to classify household objects, recognize fine textures, and infer
their physical properties both through numerical simulations and experiments
with a smart countertop prototype.

本文介绍一种新的基于视觉的传感器，并结合高分辨率触觉发射，以统一的硬件和计算体系结构进行处理；该传感器在多模式物体识别和计量方面表现出良好效果，并且在触觉和视觉领域上具有重要的物理含义，数据验证了其能力可以从深度学习视觉和触觉结合的角度对家庭物品进行分类，识别细节纹理并推断其物理属性。