In Vision-and-Language Navigation (VLN), researchers typically take an image
encoder pre-trained on ImageNet without fine-tuning on the environments that
the agent will be trained or tested on. However, the distribution shift between
the training images from ImageNet and the views in the navigation environments
may render the ImageNet pre-trained image encoder suboptimal. Therefore, in
this paper, we design a set of structure-encoding auxiliary tasks (SEA) that
leverage the data in the navigation environments to pre-train and improve the
image encoder. Specifically, we design and customize (1) 3D jigsaw, (2)
traversability prediction, and (3) instance classification to pre-train the
image encoder. Through rigorous ablations, our SEA pre-trained features are
shown to better encode structural information of the scenes, which ImageNet
pre-trained features fail to properly encode but is crucial for the target
navigation task. The SEA pre-trained features can be easily plugged into
existing VLN agents without any tuning. For example, on Test-Unseen
environments, the VLN agents combined with our SEA pre-trained features achieve
absolute success rate improvement of 12% for Speaker-Follower, 5% for
Env-Dropout, and 4% for AuxRN.

通过结构编码辅助任务 (SEA) 预先训练视觉编码器，可以提高视觉和语言导航中绝对成功率，尤其是在测试环境中，这种预训练技术可以很好地编码场景的结构信息。

结构编码辅助任务在视觉语言导航中改进视觉表示

Structure-Encoding Auxiliary Tasks for Improved Visual Representation in Vision-and-Language Navigation

Self-supervised learning in computer vision aims to pre-train an image
encoder using a large amount of unlabeled images or (image, text) pairs. The
pre-trained image encoder can then be used as a feature extractor to build
downstream classifiers for many downstream tasks with a small amount of or no
labeled training data. In this work, we propose BadEncoder, the first backdoor
attack to self-supervised learning. In particular, our BadEncoder injects
backdoors into a pre-trained image encoder such that the downstream classifiers
built based on the backdoored image encoder for different downstream tasks
simultaneously inherit the backdoor behavior. We formulate our BadEncoder as an
optimization problem and we propose a gradient descent based method to solve
it, which produces a backdoored image encoder from a clean one. Our extensive
empirical evaluation results on multiple datasets show that our BadEncoder
achieves high attack success rates while preserving the accuracy of the
downstream classifiers. We also show the effectiveness of BadEncoder using two
publicly available, real-world image encoders, i.e., Google's image encoder
pre-trained on ImageNet and OpenAI's Contrastive Language-Image Pre-training
(CLIP) image encoder pre-trained on 400 million (image, text) pairs collected
from the Internet. Moreover, we consider defenses including Neural Cleanse and
MNTD (empirical defenses) as well as PatchGuard (a provable defense). Our
results show that these defenses are insufficient to defend against BadEncoder,
highlighting the needs for new defenses against our BadEncoder. Our code is
publicly available at: this https URL

本研究研究自监督学习中的后门攻击，提出了 BadEncoder，它将后门注入预训练的图像编码器中，同时保留了下游分类器的准确性。对不同数据集的实验结果表明，这种攻击方法具有高攻击成功率，而常规的防御方法并不能有效防御。