Self supervision and natural language supervision have emerged as two
exciting ways to train general purpose image encoders which excel at a variety
of downstream tasks. Recent works such as M3AE and SLIP have suggested that
these approaches can be effectively combined, but most notably their results
use small pre-training datasets (<50M samples) and don't effectively reflect
the large-scale regime (>100M examples) that is commonly used for these
approaches. Here we investigate whether a similar approach can be effective
when trained with a much larger amount of data. We find that a combination of
two state of the art approaches: masked auto-encoders, MAE and contrastive
language image pre-training, CLIP provides a benefit over CLIP when trained on
a corpus of 11.3M image-text pairs, but little to no benefit (as evaluated on a
suite of common vision tasks) over CLIP when trained on a large corpus of 1.4B
images. Our work provides some much needed clarity into the effectiveness (or
lack thereof) of self supervision for large-scale image-text training.

本研究探讨了自监督与自然语言监督相结合，对大规模图像 - 文本训练的影响，结果表明，这种结合方式能够提高小数据集的表现，但对大规模数据集并没有显著提升。

自监督不助于大规模自然语言监督

Self Supervision Does Not Help Natural Language Supervision at Scale

How does one adapt a pre-trained visual model to novel downstream tasks
without task-specific finetuning or any model modification? Inspired by
prompting in NLP, this paper investigates visual prompting: given input-output
image example(s) of a new task at test time and a new input image, the goal is
to automatically produce the output image, consistent with the given examples.
We show that posing this problem as simple image inpainting - literally just
filling in a hole in a concatenated visual prompt image - turns out to be
surprisingly effective, provided that the inpainting algorithm has been trained
on the right data. We train masked auto-encoders on a new dataset that we
curated - 88k unlabeled figures from academic papers sources on Arxiv. We apply
visual prompting to these pretrained models and demonstrate results on various
downstream image-to-image tasks, including foreground segmentation, single
object detection, colorization, edge detection, etc.

本文通过在测试时间给定新任务的输入输出图像样例及一个新的输入图像，自动产生与这些样例一致的输出图像的方式，探讨了如何在没有特定任务的微调或任何模型修改的情况下，将预训练的视觉模型适应于新的下游任务，并证明了将图像修补算法训练在正确的数据上的效果。