Despite the advanced capabilities of contemporary machine learning (ML)
models, they remain vulnerable to adversarial and backdoor attacks. This
vulnerability is particularly concerning in real-world deployments, where
compromised models may exhibit unpredictable behavior in critical scenarios.
Such risks are heightened by the prevalent practice of collecting massive,
internet-sourced datasets for pre-training multimodal models, as these datasets
may harbor backdoors. Various techniques have been proposed to mitigate the
effects of backdooring in these models such as CleanCLIP which is the current
state-of-the-art approach.
In this work, we demonstrate that the efficacy of CleanCLIP in mitigating
backdoors is highly dependent on the particular objective used during model
pre-training.
We observe that stronger pre-training objectives correlate with harder to
remove backdoors behaviors. We show this by training multimodal models on two
large datasets consisting of 3 million (CC3M) and 6 million (CC6M) datapoints,
under various pre-training objectives, followed by poison removal using
CleanCLIP. We find that CleanCLIP is ineffective when stronger pre-training
objectives are used, even with extensive hyperparameter tuning.
Our findings underscore critical considerations for ML practitioners who
pre-train models using large-scale web-curated data and are concerned about
potential backdoor threats. Notably, our results suggest that simpler
pre-training objectives are more amenable to effective backdoor removal. This
insight is pivotal for practitioners seeking to balance the trade-offs between
using stronger pre-training objectives and security against backdoor attacks.

通过对大规模网络整理的数据进行预训练，我们发现使用简单的预训练目标更有利于有效消除后门攻击，这对于使用强预训练目标与反后门攻击之间权衡的从业者至关重要。

有效的后门缓解取决于预训练目标

Effective Backdoor Mitigation Depends on the Pre-training Objective

Multimodal contrastive pretraining has been used to train multimodal
representation models, such as CLIP, on large amounts of paired image-text
data. However, previous studies have revealed that such models are vulnerable
to backdoor attacks. Specifically, when trained on backdoored examples, CLIP
learns spurious correlations between the embedded backdoor trigger and the
target label, aligning their representations in the joint embedding space.
Injecting even a small number of poisoned examples, such as 75 examples in 3
million pretraining data, can significantly manipulate the model's behavior,
making it difficult to detect or unlearn such correlations. To address this
issue, we propose CleanCLIP, a finetuning framework that weakens the learned
spurious associations introduced by backdoor attacks by independently
re-aligning the representations for individual modalities. We demonstrate that
unsupervised finetuning using a combination of multimodal contrastive and
unimodal self-supervised objectives for individual modalities can significantly
reduce the impact of the backdoor attack. Additionally, we show that supervised
finetuning on task-specific labeled image data removes the backdoor trigger
from the CLIP vision encoder. We show empirically that CleanCLIP maintains
model performance on benign examples while erasing a range of backdoor attacks
on multimodal contrastive learning.

介绍了一种解决多模态表征学习领域中后门攻击问题的新方法 CleanCLIP，可以帮助模型减弱后门数据带来的错误训练结果。同时，该方法可以通过对单个模态的预训练任务进行微调的方式来提高模型的鲁棒性。