Although pre-training on a large amount of data is beneficial for robot
learning, current paradigms only perform large-scale pretraining for visual
representations, whereas representations for other modalities are trained from
scratch. In contrast to the abundance of visual data, it is unclear what
relevant internet-scale data may be used for pretraining other modalities such
as tactile sensing. Such pretraining becomes increasingly crucial in the
low-data regimes common in robotics applications. In this paper, we address
this gap by using contact microphones as an alternative tactile sensor. Our key
insight is that contact microphones capture inherently audio-based information,
allowing us to leverage large-scale audio-visual pretraining to obtain
representations that boost the performance of robotic manipulation. To the best
of our knowledge, our method is the first approach leveraging large-scale
multisensory pre-training for robotic manipulation. For supplementary
information including videos of real robot experiments, please see
this https URL

通过使用接触式麦克风作为替代触觉传感器，本文介绍了第一种利用大规模多模态先前训练进行机器人操作的方法，通过从大规模音频 - 视觉先前训练中获取表示来提高机器人操作的性能。

听触：面向丰富接触操控的音频 - 视觉预训练

Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulation

The Video-to-Audio (V2A) model has recently gained attention for its
practical application in generating audio directly from silent videos,
particularly in video/film production. However, previous methods in V2A have
limited generation quality in terms of temporal synchronization and
audio-visual relevance. We present Diff-Foley, a synchronized Video-to-Audio
synthesis method with a latent diffusion model (LDM) that generates
high-quality audio with improved synchronization and audio-visual relevance. We
adopt contrastive audio-visual pretraining (CAVP) to learn more temporally and
semantically aligned features, then train an LDM with CAVP-aligned visual
features on spectrogram latent space. The CAVP-aligned features enable LDM to
capture the subtler audio-visual correlation via a cross-attention module. We
further significantly improve sample quality with `double guidance'. Diff-Foley
achieves state-of-the-art V2A performance on current large scale V2A dataset.
Furthermore, we demonstrate Diff-Foley practical applicability and
generalization capabilities via downstream finetuning. Project Page: see
this https URL

使用 Diff-Foley 方法和潜在扩散模型进行音频与视频同步生成，其在现有的大规模数据集上取得了最先进的性能。