Humans rely on their visual and tactile senses to develop a comprehensive 3D
understanding of their physical environment. Recently, there has been a growing
interest in exploring and manipulating objects using data-driven approaches
that utilise high-resolution vision-based tactile sensors. However, 3D shape
reconstruction using tactile sensing has lagged behind visual shape
reconstruction because of limitations in existing techniques, including the
inability to generalise over unseen shapes, the absence of real-world testing,
and limited expressive capacity imposed by discrete representations. To address
these challenges, we propose TouchSDF, a Deep Learning approach for tactile 3D
shape reconstruction that leverages the rich information provided by a
vision-based tactile sensor and the expressivity of the implicit neural
representation DeepSDF. Our technique consists of two components: (1) a
Convolutional Neural Network that maps tactile images into local meshes
representing the surface at the touch location, and (2) an implicit neural
function that predicts a signed distance function to extract the desired 3D
shape. This combination allows TouchSDF to reconstruct smooth and continuous 3D
shapes from tactile inputs in simulation and real-world settings, opening up
research avenues for robust 3D-aware representations and improved multimodal
perception in robotics. Code and supplementary material are available at:
this https URL

通过利用基于视觉的触觉传感器提供的丰富信息和 DeepSDF 的内在神经表示表达能力，我们提出了 TouchSDF，一种用于触觉 3D 形状重建的深度学习方法。这种方法包括两个组件：（1）将触觉图像映射到表示触摸位置表面的局部网格的卷积神经网络，以及（2）预测有符号距离函数的隐式神经函数，以提取所需的 3D 形状。这种组合使得 TouchSDF 能够在模拟和真实世界环境中，从触觉输入中重建出平滑连续的 3D 形状，为机器人学中的稳健 3D 感知和改进多模态感知技术开辟了研究途径。

TouchSDF：基于视觉触觉传感的三维形状重建的 DeepSDF 方法

TouchSDF: A DeepSDF Approach for 3D Shape Reconstruction using  Vision-Based Tactile Sensing

In this paper, with the goal of quantifying the qualitative image outputs of
a Vision-based Tactile Sensor (VTS), we present the design, fabrication, and
characterization of a novel Quantitative Surface Tactile Sensor (called QS-TS).
QS-TS directly estimates the sensor's gel layer deformation in real-time
enabling safe and autonomous tactile manipulation and servoing of delicate
objects using robotic manipulators. The core of the proposed sensor is the
utilization of miniature 1.5 mm x 1.5 mm synthetic square markers with inner
binary patterns and a broad black border, called ArUco Markers. Each ArUco
marker can provide real-time camera pose estimation that, in our design, is
used as a quantitative measure for obtaining deformation of the QS-TS gel
layer. Moreover, thanks to the use of ArUco markers, we propose a unique
fabrication procedure that mitigates various challenges associated with the
fabrication of the existing marker-based VTSs and offers an intuitive and
less-arduous method for the construction of the VTS. Remarkably, the proposed
fabrication facilitates the integration and adherence of markers with the gel
layer to robustly and reliably obtain a quantitative measure of deformation in
real-time regardless of the orientation of ArUco Markers. The performance and
efficacy of the proposed QS-TS in estimating the deformation of the sensor's
gel layer were experimentally evaluated and verified. Results demonstrate the
phenomenal performance of the QS-TS in estimating the deformation of the gel
layer with a relative error of <5%.

本文介绍了一种新型量化表面触觉传感器（称为 QS-TS），该传感器利用 ArUco 标记实时估计凝胶层的变形，能够安全和自主地使用机器人操纵器进行触觉操作和伺服精细物体。实验结果展示了 QS-TS 在估计凝胶层变形方面出色的性能，相对误差小于 5%。