We introduce Gaussian masking for Language-Image Pre-Training (GLIP) a novel,
straightforward, and effective technique for masking image patches during
pre-training of a vision-language model. GLIP builds on Fast Language-Image
Pre-Training (FLIP), which randomly masks image patches while training a CLIP
model. GLIP replaces random masking with centered masking, that uses a Gaussian
distribution and is inspired by the importance of image patches at the center
of the image. GLIP retains the same computational savings as FLIP, while
improving performance across a range of downstream datasets and tasks, as
demonstrated by our experimental results. We show the benefits of GLIP to be
easy to obtain, requiring no delicate tuning of the Gaussian, and also
applicable to data sets containing images without an obvious center focus.

我们介绍了用于语言 - 图像预训练的高斯掩模化 (GLIP) 技术，它是一种新颖、直接且有效的技术，在预训练视觉 - 语言模型期间屏蔽图像补丁。GLIP 基于快速语言 - 图像预训练 (FLIP)，在训练 CLIP 模型时随机屏蔽图像补丁。GLIP 用高斯分布替换了随机屏蔽，并受到图像中心补丁的重要性启发。GLIP 与 FLIP 具有相同的计算节约效果，并通过我们的实验结果在一系列下游数据集和任务中改善了性能。我们展示了获得 GLIP 的好处容易，不需要对高斯进行精细调整，并适用于包含没有明显中心焦点的图像数据集。

面向语言 - 图像预训练的居中掩蔽

Centered Masking for Language-Image Pre-Training

With the limited availability of labeled data with various atmospheric
conditions in remote sensing images, it seems useful to work with
self-supervised algorithms. Few pretext-based algorithms, including from
rotation, spatial context and jigsaw puzzles are not appropriate for satellite
images. Often, satellite images have a higher temporal frequency. So, the
temporal dimension of remote sensing data provides natural augmentation without
requiring us to create artificial augmentation of images. Here, we propose
S3-TSS, a novel method of self-supervised learning technique that leverages
natural augmentation occurring in temporal dimension. We compare our results
with current state-of-the-art methods and also perform various experiments. We
observed that our method was able to perform better than baseline SeCo in four
downstream datasets. Code for our work can be found here:
this https URL

提出了一种名为 S3-TSS 的自监督学习技术方法，利用时序维度中自然增强的遥感数据进行训练，在四个下游数据集中表现优于基准算法 SeCo。

卫星图像的时间自监督（S3-TSS）：一种卫星图像中的 SSL 技术的新方法

Self-Supervision in Time for Satellite Images(S3-TSS): A novel method of  SSL technique in Satellite images

For most natural language processing tasks, the dominant practice is to
finetune large pretrained transformer models (e.g., BERT) using smaller
downstream datasets. Despite the success of this approach, it remains unclear
to what extent these gains are attributable to the massive background corpora
employed for pretraining versus to the pretraining objectives themselves. This
paper introduces a large-scale study of self-pretraining, where the same
(downstream) training data is used for both pretraining and finetuning. In
experiments addressing both ELECTRA and RoBERTa models and 10 distinct
downstream datasets, we observe that self-pretraining rivals standard
pretraining on the BookWiki corpus (despite using around
$10\times$--$500\times$ less data), outperforming the latter on $7$ and $5$
datasets, respectively. Surprisingly, these task-specific pretrained models
often perform well on other tasks, including the GLUE benchmark. Our results
suggest that in many scenarios, performance gains attributable to pretraining
are driven primarily by the pretraining objective itself and are not always
attributable to the incorporation of massive datasets. These findings are
especially relevant in light of concerns about intellectual property and
offensive content in web-scale pretraining data.

本文介绍了一个大规模的自我训练研究，其中使用相同的（下游）训练数据进行预训练和微调，并且观察到自我预训练可以与标准预训练相媲美，这表明在许多情况下，预训练性能增益主要受预训练目标本身的驱动，而不一定是庞大数据集的影响。