Contrastive learning has emerged as a powerful technique in audio-visual representation learning, leveraging the natural co-occurrence of audio and visual modalities in extensive web-scale video datasets to achieve significant advancements. However, conventional contrastive audio-visual learning methodologies often rely on aggregated representations derived through temporal aggregation, which neglects the intrinsic sequential nature of the data. This oversight raises concerns regarding the ability of standard approaches to capture and utilize fine-grained information within sequences, information that is vital for distinguishing between semantically similar yet distinct examples. In response to this limitation, we propose sequential contrastive audio-visual learning (SCAV), which contrasts examples based on their non-aggregated representation space using sequential distances. Retrieval experiments with the VGGSound and Music datasets demonstrate the effectiveness of SCAV, showing 2-3x relative improvements against traditional aggregation-based contrastive learning and other methods from the literature. We also show that models trained with SCAV exhibit a high degree of flexibility regarding the metric employed for retrieval, allowing them to operate on a spectrum of efficiency-accuracy trade-offs, potentially making them applicable in multiple scenarios, from small- to large-scale retrieval.

基于时间聚合派生的合并表示方式对音频-视觉对比学习的局限性提出关注，因此提出了顺序对比音频-视觉学习 (SCAV)方法，在非合并表示空间中使用顺序距离对示例进行对比。使用VGGSound和Music数据集进行检索实验验证了SCAV的有效性，相对于传统的基于聚合的对比学习和文献中的其他方法，SCAV取得了2-3倍的相对改进。同时，还展示了使用SCAV训练的模型在检索时对所采用的度量具有高度的灵活性，可以在效率-准确性的多个场景中应用。

序列对比视听学习