The ultimate goal for foundation models is realizing task-agnostic, i.e.,
supporting out-of-the-box usage without task-specific fine-tuning. Although
breakthroughs have been made in natural language processing and image
representation learning, it is still challenging for video models to reach it
due to the increasing uncertainty of spatiotemporal signals. To ease training,
existing works leverage image foundation models' prior knowledge and equip them
with efficient temporal modules. Despite the satisfactory fine-tuning
performance, we empirically find they fall short of out-of-the-box usage, given
the even degraded performance in zero-shot/linear protocols compared to their
baseline counterparts. In this work, we analyze the factor that leads to
degradation from the perspective of language supervision distortion. We argue
that tuning a text encoder end-to-end, as done in previous work, is suboptimal
since it may overfit in terms of styles, thereby losing its original
generalization ability to capture the semantics of various language registers.
The overfitted text encoder, in turn, provides a harmful supervision signal,
degrading the video representation. To tackle this issue, we propose a
degradation-free pre-training strategy to retain the generalization ability of
the text encoder via freezing shallow layers while enabling the task-related
semantics capturing in tunable deep layers. As for the training objective, we
adopted the transcript sorting task in TVTS incorporated with masking
techniques to enable scalable training. As a result, we produce a series of
models, dubbed TVTSv2, with up to one billion parameters. We achieve new
state-of-the-arts on various video benchmarks with a frozen backbone,
surpassing the recent ImageBind, InternVideo, etc. Code is available at
this https URL

本论文分析了导致视频模型性能下降的因素 —— 语言监督失真，提出了一种去除降级的预训练策略，并采用排序任务同时使用掩蔽技术进行可扩展的训练，得到了一系列新的模型。