Self-supervised learning (SSL) for RGB images has achieved significant
success, yet there is still limited research on SSL for infrared images,
primarily due to three prominent challenges: 1) the lack of a suitable
large-scale infrared pre-training dataset, 2) the distinctiveness of non-iconic
infrared images rendering common pre-training tasks like masked image modeling
(MIM) less effective, and 3) the scarcity of fine-grained textures making it
particularly challenging to learn general image features. To address these
issues, we construct a Multi-Scene Infrared Pre-training (MSIP) dataset
comprising 178,756 images, and introduce object-sensitive random RoI cropping,
an image preprocessing method, to tackle the challenge posed by non-iconic
images. To alleviate the impact of weak textures on feature learning, we
propose a pre-training paradigm called Pre-training with ADapter (PAD), which
uses adapters to learn domain-specific features while freezing parameters
pre-trained on ImageNet to retain the general feature extraction capability.
This new paradigm is applicable to any transformer-based SSL method.
Furthermore, to achieve more flexible coordination between pre-trained and
newly-learned features in different layers and patches, a patchwise-scale
adapter with dynamically learnable scale factors is introduced. Extensive
experiments on three downstream tasks show that PAD, with only 1.23M
pre-trainable parameters, outperforms other baseline paradigms including
continual full pre-training on MSIP. Our code and dataset are available at
this https URL

该研究介绍了自主学习在红外图像领域的应用，提供了一个大规模红外预训练数据集，通过对象敏感的随机 RoI 裁剪方法和面向纹理缺失的图像预处理方法解决了传统方法在红外图像上的挑战，并引入可动态学习尺度因子的分块适配器来实现预训练特征和新特征之间的灵活协调。实验证明，在三个下游任务上，该方法优于其他基线模型，仅使用了 1.23M 可预训练参数。

PAD：自监督预训练与红外图像的 Patchwise-Scale Adapter

PAD: Self-Supervised Pre-Training with Patchwise-Scale Adapter for  Infrared Images

In this work, we present Auto-captions on GIF, which is a new large-scale
pre-training dataset for generic video understanding. All video-sentence pairs
are created by automatically extracting and filtering video caption annotations
from billions of web pages. Auto-captions on GIF dataset can be utilized to
pre-train the generic feature representation or encoder-decoder structure for
video captioning, and other downstream tasks (e.g., sentence localization in
videos, video question answering, etc.) as well. We present a detailed analysis
of Auto-captions on GIF dataset in comparison to existing video-sentence
datasets. We also provide an evaluation of a Transformer-based encoder-decoder
structure for vision-language pre-training, which is further adapted to video
captioning downstream task and yields the compelling generalizability on
MSR-VTT. The dataset is available at
http://www.auto-video-captions.top/2020/dataset.

Auto-captions on GIF 是一个新的大规模预训练数据集，可用于视频理解、视频字幕生成等下游任务的通用特征表示或编码器 - 解码器结构的预训练。通过 Auto-captions on GIF 数据集，我们还对基于 Transformer 的编码器 - 解码器结构进行了分析和评估，并在 MSR-VTT 上获得了强大的泛化性能。