Recent contrastive language image pre-training has led to learning highly
transferable and robust image representations. However, adapting these models
to video domains with minimal supervision remains an open problem. We explore a
simple step in that direction, using language tied self-supervised learning to
adapt an image CLIP model to the video domain. A backbone modified for temporal
modeling is trained under self-distillation settings with train objectives
operating in an action concept space. Feature vectors of various action
concepts extracted from a language encoder using relevant textual prompts
construct this space. We introduce two train objectives, concept distillation
and concept alignment, that retain generality of original representations while
enforcing relations between actions and their attributes. Our approach improves
zero-shot and linear probing performance on three action recognition
benchmarks.

使用自监督学习方法，对图像 CLIP 模型进行语言约束的调整，以适应视频领域，提高三个行动识别基准测试的零样本和线性探测性能。

基于语言的动作概念空间改进视频自监督学习

Language-based Action Concept Spaces Improve Video Self-Supervised  Learning

Self supervision and natural language supervision have emerged as two
exciting ways to train general purpose image encoders which excel at a variety
of downstream tasks. Recent works such as M3AE and SLIP have suggested that
these approaches can be effectively combined, but most notably their results
use small pre-training datasets (<50M samples) and don't effectively reflect
the large-scale regime (>100M examples) that is commonly used for these
approaches. Here we investigate whether a similar approach can be effective
when trained with a much larger amount of data. We find that a combination of
two state of the art approaches: masked auto-encoders, MAE and contrastive
language image pre-training, CLIP provides a benefit over CLIP when trained on
a corpus of 11.3M image-text pairs, but little to no benefit (as evaluated on a
suite of common vision tasks) over CLIP when trained on a large corpus of 1.4B
images. Our work provides some much needed clarity into the effectiveness (or
lack thereof) of self supervision for large-scale image-text training.

本研究探讨了自监督与自然语言监督相结合，对大规模图像 - 文本训练的影响，结果表明，这种结合方式能够提高小数据集的表现，但对大规模数据集并没有显著提升。