We introduce VideoPrism, a general-purpose video encoder that tackles diverse
video understanding tasks with a single frozen model. We pretrain VideoPrism on
a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M
video clips with noisy parallel text (e.g., ASR transcripts). The pretraining
approach improves upon masked autoencoding by global-local distillation of
semantic video embeddings and a token shuffling scheme, enabling VideoPrism to
focus primarily on the video modality while leveraging the invaluable text
associated with videos. We extensively test VideoPrism on four broad groups of
video understanding tasks, from web video question answering to CV for science,
achieving state-of-the-art performance on 30 out of 33 video understanding
benchmarks.

介绍了一种名为 VideoPrism 的通用视频编码器，利用单个冻结模型解决各种视频理解任务。通过在异构语料库上预训练 VideoPrism，其中包含 3600 万个高质量的视频字幕对和 5.82 亿个带有噪声的平行文本视频剪辑，通过全局 - 局部语义视频嵌入和标记重排方案改进了遮挡自编码模型的预训练方法，使得 VideoPrism 主要关注视频模式，同时利用与视频相关的宝贵文本。我们在四个广泛的视频理解任务组上对 VideoPrism 进行了广泛的测试，从网络视频问答到科学计算机视觉，实现了 30 个有 33 个视频理解基准中的最优性能。