Over the past several years, the synchronization between audio and visual
signals has been leveraged to learn richer audio-visual representations. Aided
by the large availability of unlabeled videos, many unsupervised training
frameworks have demonstrated impressive results in various downstream audio and
video tasks. Recently, Masked Audio-Video Learners (MAViL) has emerged as a
state-of-the-art audio-video pre-training framework. MAViL couples contrastive
learning with masked autoencoding to jointly reconstruct audio spectrograms and
video frames by fusing information from both modalities. In this paper, we
study the potential synergy between diffusion models and MAViL, seeking to
derive mutual benefits from these two frameworks. The incorporation of
diffusion into MAViL, combined with various training efficiency methodologies
that include the utilization of a masking ratio curriculum and adaptive batch
sizing, results in a notable 32% reduction in pre-training Floating-Point
Operations (FLOPS) and an 18% decrease in pre-training wall clock time.
Crucially, this enhanced efficiency does not compromise the model's performance
in downstream audio-classification tasks when compared to MAViL's performance.

学习了音频和视觉信号的同步已经被利用来学习更丰富的音频 - 视觉表示，最新的研究结合了扩散模型与音频 - 视频预训练框架 MAViL，通过使用掩蔽编码和对比学习来实现音频谱图和视频帧的联合重构，并通过训练效率方法的结合降低了 32％的浮点预训练操作数量 (FLOPS) 和 18％的预训练时间，与 MAViL 相比，在下游音频分类任务上没有损害模型的性能。