We introduce LAVITI, a novel approach to learning language, video, and
temporal representations in long-form videos via contrastive learning.
Different from pre-training on video-text pairs like EgoVLP, LAVITI aims to
align language, video, and temporal features by extracting meaningful moments
in untrimmed videos. Our model employs a set of learnable moment queries to
decode clip-level visual, language, and temporal features. In addition to
vision and language alignment, we introduce relative temporal embeddings (TE)
to represent timestamps in videos, which enables contrastive learning of time.
Significantly different from traditional approaches, the prediction of a
particular timestamp is transformed by computing the similarity score between
the predicted TE and all TEs. Furthermore, existing approaches for video
understanding are mainly designed for short videos due to high computational
complexity and memory footprint. Our method can be trained on the Ego4D dataset
with only 8 NVIDIA RTX-3090 GPUs in a day. We validated our method on
CharadesEgo action recognition, achieving state-of-the-art results.

通过对比学习，LAVITI 引入了一种新方法来学习长视频中的语言、视觉和时间表示，目标是通过提取未修剪视频中的有意义的时刻来对齐语言、视频和时间特征，并使用可学习的时刻查询来解码视频和时间特征。此外，引入了相对时间嵌入（TE）来表示视频中的时间戳，从而实现时间的对比学习。该方法在计算复杂度和内存占用方面显著不同于传统方法，可以仅使用 8 个 NVIDIA RTX-3090 GPU 在一天内在 Ego4D 数据集上进行训练。在 CharadesEgo 动作识别任务上，我们通过验证该方法实现了最先进的结果。