We present a scene representation, which we call a tactile-augmented radiance
field (TaRF), that brings vision and touch into a shared 3D space. This
representation can be used to estimate the visual and tactile signals for a
given 3D position within a scene. We capture a scene's TaRF from a collection
of photos and sparsely sampled touch probes. Our approach makes use of two
insights: (i) common vision-based touch sensors are built on ordinary cameras
and thus can be registered to images using methods from multi-view geometry,
and (ii) visually and structurally similar regions of a scene share the same
tactile features. We use these insights to register touch signals to a captured
visual scene, and to train a conditional diffusion model that, provided with an
RGB-D image rendered from a neural radiance field, generates its corresponding
tactile signal. To evaluate our approach, we collect a dataset of TaRFs. This
dataset contains more touch samples than previous real-world datasets, and it
provides spatially aligned visual signals for each captured touch signal. We
demonstrate the accuracy of our cross-modal generative model and the utility of
the captured visual-tactile data on several downstream tasks. Project page:
this https URL

我们提出了一种场景表示，称为触觉增强辐射场（TaRF），将视觉和触觉结合到共享的 3D 空间中，通过估计场景中给定 3D 位置的视觉和触觉信号。我们通过使用多视角几何方法将触觉信号注册到捕获的视觉场景上，并训练了一个条件扩散模型，通过从神经辐射场渲染的 RGB-D 图像生成其相应的触觉信号。通过收集包含较多触觉样本的 TaRF 数据集，并为每个捕获的触觉信号提供空间对齐的视觉信号，我们评估了我们的方法的准确性和捕获的视觉 - 触觉数据在几个下游任务上的实用性。