We introduce VideoPrism, a general-purpose video encoder that tackles diverse
video understanding tasks with a single frozen model. We pretrain VideoPrism on
a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M
video clips with noisy parallel text (e.g., ASR transcripts). The pretraining
approach improves upon masked autoencoding by global-local distillation of
semantic video embeddings and a token shuffling scheme, enabling VideoPrism to
focus primarily on the video modality while leveraging the invaluable text
associated with videos. We extensively test VideoPrism on four broad groups of
video understanding tasks, from web video question answering to CV for science,
achieving state-of-the-art performance on 30 out of 33 video understanding
benchmarks.

介绍了一种名为 VideoPrism 的通用视频编码器，利用单个冻结模型解决各种视频理解任务。通过在异构语料库上预训练 VideoPrism，其中包含 3600 万个高质量的视频字幕对和 5.82 亿个带有噪声的平行文本视频剪辑，通过全局 - 局部语义视频嵌入和标记重排方案改进了遮挡自编码模型的预训练方法，使得 VideoPrism 主要关注视频模式，同时利用与视频相关的宝贵文本。我们在四个广泛的视频理解任务组上对 VideoPrism 进行了广泛的测试，从网络视频问答到科学计算机视觉，实现了 30 个有 33 个视频理解基准中的最优性能。

VideoPrism：一种用于视频理解的基础视觉编码器

VideoPrism: A Foundational Visual Encoder for Video Understanding

Medical image classification is one of the most important tasks for
computer-aided diagnosis. Deep learning models, particularly convolutional
neural networks, have been successfully used for disease classification from
medical images, facilitated by automated feature learning. However, the diverse
imaging modalities and clinical pathology make it challenging to construct
generalized and robust classifications. Towards improving the model
performance, we propose a novel pretraining approach, namely Forward Forward
Contrastive Learning (FFCL), which leverages the Forward-Forward Algorithm in a
contrastive learning framework--both locally and globally. Our experimental
results on the chest X-ray dataset indicate that the proposed FFCL achieves
superior performance (3.69% accuracy over ImageNet pretrained ResNet-18) over
existing pretraining models in the pneumonia classification task. Moreover,
extensive ablation experiments support the particular local and global
contrastive pretraining design in FFCL.

该研究论文提出了一种基于前向前向对比学习的图像分类预训练方法，通过将前向算法与对比学习结合，可以构建出更加具有鲁棒性的分类模型，实验结果表明该方法可以取得比现有预训练模型更加出色的分类性能。

前向 - 前向对比学习

Forward-Forward Contrastive Learning

We introduce a new pretraining approach geared for multi-document language
modeling, incorporating two key ideas into the masked language modeling
self-supervised objective. First, instead of considering documents in
isolation, we pretrain over sets of multiple related documents, encouraging the
model to learn cross-document relationships. Second, we improve over recent
long-range transformers by introducing dynamic global attention that has access
to the entire input to predict masked tokens. We release CDLM (Cross-Document
Language Model), a new general language model for multi-document setting that
can be easily applied to downstream tasks. Our extensive analysis shows that
both ideas are essential for the success of CDLM, and work in synergy to set
new state-of-the-art results for several multi-text tasks. Code and models are
available at this https URL

该研究提出了一种新的预训练方法，针对多文档语言建模，包括两个关键思想：一是通过预训练多个相关文档集合，鼓励模型学习跨文档关系；二是通过引入动态全局注意力，改善最近的长距离 Transformer，以访问整个输入来预测掩码标记。 CDLM 是一个新的通用的多文档语言模型，可以轻松应用于下游任务。