Pretraining egocentric vision-language models has become essential to
improving downstream egocentric video-text tasks. These egocentric foundation
models commonly use the transformer architecture. The memory footprint of these
models during pretraining can be substantial. Therefore, we pretrain SViTT-Ego,
the first sparse egocentric video-text transformer model integrating edge and
node sparsification. We pretrain on the EgoClip dataset and incorporate the
egocentric-friendly objective EgoNCE, instead of the frequently used InfoNCE.
Most notably, SViTT-Ego obtains a +2.8% gain on EgoMCQ (intra-video) accuracy
compared to LAVILA large, with no additional data augmentation techniques other
than standard image augmentations, yet pretrainable on memory-limited devices.

通过在 EgoClip 数据集上预训练，采用稀疏自我中心视频文本变换模型 SViTT-Ego，融合了边缘和节点稀疏化，以及友好的自我中心目标 EgoNCE，相对于 LAVILA large，在内部视频 EgoMCQ 上获得了 + 2.8% 的准确度提升，无需额外的数据增强技术，可以在内存有限的设备上预训练。

SViTT-Ego：一种用于自我中心视频的稀疏视频文本变压器

SViTT-Ego: A Sparse Video-Text Transformer for Egocentric Video

Video-Language Pretraining (VLP), which aims to learn transferable
representation to advance a wide range of video-text downstream tasks, has
recently received increasing attention. Best performing works rely on
large-scale, 3rd-person video-text datasets, such as HowTo100M. In this work,
we exploit the recently released Ego4D dataset to pioneer Egocentric VLP along
three directions. (i) We create EgoClip, a 1st-person video-text pretraining
dataset comprising 3.8M clip-text pairs well-chosen from Ego4D, covering a
large variety of human daily activities. (ii) We propose a novel pretraining
objective, dubbed EgoNCE, which adapts video-text contrastive learning to the
egocentric domain by mining egocentric-aware positive and negative samples.
(iii) We introduce EgoMCQ, a development benchmark that is close to EgoClip and
hence can support effective validation and fast exploration of our design
decisions in EgoClip and EgoNCE. Furthermore, we demonstrate strong performance
on five egocentric downstream tasks across three datasets: video-text retrieval
on EPIC-KITCHENS-100; action recognition on Charades-Ego; natural language
query, moment query, and object state change classification on Ego4D challenge
benchmarks. The dataset and code are available at
this https URL.

本文提出了一种新的视频语言预训练（Video-Language Pretraining）方法 ——Egocentric VLP，通过利用新发布的 Ego4D 数据集，分别从 EgoClip，EgoNCE 和 EgoMCQ 三个方向进行探索，最终成功在五个自我中心任务上实现了强大的表现。