This paper presents T3: Transferable Tactile Transformers, a framework for
tactile representation learning that scales across multi-sensors and
multi-tasks. T3 is designed to overcome the contemporary issue that
camera-based tactile sensing is extremely heterogeneous, i.e. sensors are built
into different form factors, and existing datasets were collected for disparate
tasks. T3 captures the shared latent information across different sensor-task
pairings by constructing a shared trunk transformer with sensor-specific
encoders and task-specific decoders. The pre-training of T3 utilizes a novel
Foundation Tactile (FoTa) dataset, which is aggregated from several
open-sourced datasets and it contains over 3 million data points gathered from
13 sensors and 11 tasks. FoTa is the largest and most diverse dataset in
tactile sensing to date and it is made publicly available in a unified format.
Across various sensors and tasks, experiments show that T3 pre-trained with
FoTa achieved zero-shot transferability in certain sensor-task pairings, can be
further fine-tuned with small amounts of domain-specific data, and its
performance scales with bigger network sizes. T3 is also effective as a tactile
encoder for long horizon contact-rich manipulation. Results from sub-millimeter
multi-pin electronics insertion tasks show that T3 achieved a task success rate
25% higher than that of policies trained with tactile encoders trained from
scratch, or 53% higher than without tactile sensing. Data, code, and model
checkpoints are open-sourced at this https URL

本研究提出了 T3：可传输触觉变压器，这是一个能够在多个传感器和多个任务之间进行扩展的触觉表示学习框架。T3 通过构建共享的主干变压器与传感器特定的编码器和任务特定的解码器来捕捉不同传感器任务对之间的共享潜在信息。使用 FoTa 数据集进行的预训练表明，T3 在某些传感器 - 任务对之间实现了零 - shot 可传输性，并且可以通过少量特定领域数据进行进一步微调，而且其性能随着网络规模的增大而扩展。T3 还作为长时间接触丰富型操纵的触觉编码器具有很好的效果。