In this paper, we consider the problem of audio-visual synchronisation
applied to videos `in-the-wild' (ie of general classes beyond speech). As a new
task, we identify and curate a test set with high audio-visual correlation,
namely VGG-Sound Sync. We compare a number of transformer-based architectural
variants specifically designed to model audio and visual signals of arbitrary
length, while significantly reducing memory requirements during training. We
further conduct an in-depth analysis on the curated dataset and define an
evaluation metric for open domain audio-visual synchronisation. We apply our
method on standard lip reading speech benchmarks, LRS2 and LRS3, with ablations
on various aspects. Finally, we set the first benchmark for general
audio-visual synchronisation with over 160 diverse classes in the new VGG-Sound
Sync video dataset. In all cases, our proposed model outperforms the previous
state-of-the-art by a significant margin.

本研究提出基于 transformer 的架构和度量标准用于评估各种类别下的音频 - 视频同步，并使用新的 VGG-Sound Sync 数据集测试。结果表明，我们的模型优于先前的最先进技术。