Deep neural networks are vulnerable to adversarial attacks. We consider
adversarial defense in the case of zero-shot image classification setting,
which has rarely been explored because both adversarial defense and zero-shot
learning are challenging. We propose LAAT, a novel Language-driven,
Anchor-based Adversarial Training strategy, to improve the adversarial
robustness in a zero-shot setting. LAAT uses a text encoder to obtain fixed
anchors (normalized feature embeddings) of each category, then uses these
anchors to perform adversarial training. The text encoder has the property that
semantically similar categories can be mapped to neighboring anchors in the
feature space. By leveraging this property, LAAT can make the image model
adversarially robust on novel categories without any extra examples.
Experimental results show that our method achieves impressive zero-shot
adversarial performance, even surpassing the previous state-of-the-art
adversarially robust one-shot methods in most attacking settings. When models
are trained with LAAT on large datasets like ImageNet-1K, they can have
substantial zero-shot adversarial robustness across several downstream
datasets.

本研究提出了一种新颖的基于语言驱动和锚点的对抗训练策略 LAAT，通过利用文本编码器生成锚点来提升零样本图像分类的对抗鲁棒性，实验证明 LAAT 策略达到了令人印象深刻的零样本的对抗性能，并且在大型数据集中训练的模型具有显着的零样本对抗性鲁棒性。