Robust frame-wise embeddings are essential to perform video analysis and understanding tasks. We present a self-supervised method for representation learning based on aligning temporal video sequences. Our framework uses a transformer-based encoder to extract frame-level features and leverages them to find the optimal alignment path between video sequences. We introduce the novel Local-Alignment Contrastive (LAC) loss, which combines a differentiable local alignment loss to capture local temporal dependencies with a contrastive loss to enhance discriminative learning. Prior works on video alignment have focused on using global temporal ordering across sequence pairs, whereas our loss encourages identifying the best-scoring subsequence alignment. LAC uses the differentiable Smith-Waterman (SW) affine method, which features a flexible parameterization learned through the training phase, enabling the model to adjust the temporal gap penalty length dynamically. Evaluations show that our learned representations outperform existing state-of-the-art approaches on action recognition tasks.

该研究解决了视频分析中对稳健帧级嵌入的需求，提出了一种自监督的表示学习方法，侧重于时间视频序列的对齐。通过引入局部对齐对比损失（LAC），结合了可微局部对齐损失和对比损失，显著提升了模型在动作识别任务中的表现，展现了新的对齐方法的潜力。

基于可微局部对齐的自监督视频对比学习