With the evolution of self-supervised learning, the pre-training paradigm has
emerged as a predominant solution within the deep learning landscape. Model
providers furnish pre-trained encoders designed to function as versatile
feature extractors, enabling downstream users to harness the benefits of
expansive models with minimal effort through fine-tuning. Nevertheless, recent
works have exposed a vulnerability in pre-trained encoders, highlighting their
susceptibility to downstream-agnostic adversarial examples (DAEs) meticulously
crafted by attackers. The lingering question pertains to the feasibility of
fortifying the robustness of downstream models against DAEs, particularly in
scenarios where the pre-trained encoders are publicly accessible to the
attackers.
In this paper, we initially delve into existing defensive mechanisms against
adversarial examples within the pre-training paradigm. Our findings reveal that
the failure of current defenses stems from the domain shift between
pre-training data and downstream tasks, as well as the sensitivity of encoder
parameters. In response to these challenges, we propose Genetic
Evolution-Nurtured Adversarial Fine-tuning (Gen-AF), a two-stage adversarial
fine-tuning approach aimed at enhancing the robustness of downstream models.
Our extensive experiments, conducted across ten self-supervised training
methods and six datasets, demonstrate that Gen-AF attains high testing accuracy
and robust testing accuracy against state-of-the-art DAEs.

通过采用遗传进化培育的对抗微调方法（Gen-AF），本文提出了一种两阶段的对抗微调方法，旨在增强下游模型的鲁棒性，实验证明该方法在十种自监督训练方法和六个数据集上对最先进的对抗示例具有高测试准确性和鲁棒测试准确性。

针对对抗样本安全地微调预训练编码器

Securely Fine-tuning Pre-trained Encoders Against Adversarial Examples

Most language models (LMs) are trained and applied in an autoregressive
left-to-right fashion, assuming that the next token only depends on the
preceding ones. However, this assumption ignores the potential benefits of
using the full sequence information during training, and the possibility of
having context from both sides during inference. In this paper, we propose a
new pre-training paradigm with techniques that jointly improve the training
data efficiency and the capabilities of the LMs in the infilling task. The
first is a training objective that aligns the predictions of a left-to-right LM
with those of a right-to-left LM, trained on the same data but in reverse
order. The second is a bidirectional inference procedure that enables both LMs
to meet in the middle. We show the effectiveness of our pre-training paradigm
with extensive experiments on both programming and natural language models,
outperforming strong baselines.

本文提出了一种新的预训练范例，采用双向推理和训练效率优化技术，以提高语言模型在数据填充任务中的能力并展开大量编程和自然语言实验取得优异成果。

中途相遇：一种新的预训练范式

Meet in the Middle: A New Pre-training Paradigm

How to learn discriminative video representation from unlabeled videos is
challenging but crucial for video analysis. The latest attempts seek to learn a
representation model by predicting the appearance contents in the masked
regions. However, simply masking and recovering appearance contents may not be
sufficient to model temporal clues as the appearance contents can be easily
reconstructed from a single frame. To overcome this limitation, we present
Masked Motion Encoding (MME), a new pre-training paradigm that reconstructs
both appearance and motion information to explore temporal clues. In MME, we
focus on addressing two critical challenges to improve the representation
performance: 1) how to well represent the possible long-term motion across
multiple frames; and 2) how to obtain fine-grained temporal clues from sparsely
sampled videos. Motivated by the fact that human is able to recognize an action
by tracking objects' position changes and shape changes, we propose to
reconstruct a motion trajectory that represents these two kinds of change in
the masked regions. Besides, given the sparse video input, we enforce the model
to reconstruct dense motion trajectories in both spatial and temporal
dimensions. Pre-trained with our MME paradigm, the model is able to anticipate
long-term and fine-grained motion details. Code is available at
this https URL

提出了一种名为 MME 的预训练模型，旨在通过重建外观和运动信息来探索时序线索，从而改善视频表示性能。方法着重解决多帧间的长期运动和从稀疏视频中获取细粒度时序线索这两个关键问题，并通过重建模糊区域中代表位置和形状变化的运动轨迹，使预训练模型进一步预测运动细节。