The performance of deep models, including Vision Transformers, is known to be
vulnerable to adversarial attacks. Many existing defenses against these
attacks, such as adversarial training, rely on full-model fine-tuning to induce
robustness in the models. These defenses require storing a copy of the entire
model, that can have billions of parameters, for each task. At the same time,
parameter-efficient prompt tuning is used to adapt large transformer-based
models to downstream tasks without the need to save large copies. In this
paper, we examine parameter-efficient prompt tuning of Vision Transformers for
downstream tasks under the lens of robustness. We show that previous
adversarial defense methods, when applied to the prompt tuning paradigm, suffer
from gradient obfuscation and are vulnerable to adaptive attacks. We introduce
ADAPT, a novel framework for performing adaptive adversarial training in the
prompt tuning paradigm. Our method achieves competitive robust accuracy of ~40%
w.r.t. SOTA robustness methods using full-model fine-tuning, by tuning only ~1%
of the number of parameters.

使用 ADAPT 框架对 Vision Transformers 进行参数高效提示调优，实现对下游任务的鲁棒性训练，从而在只调优了约 1% 的参数数量的情况下，实现与完全模型微调相当的鲁棒准确率（约 40%）.

增强 Prompt 调参视觉 Transformer 的适应性

ADAPT to Robustify Prompt Tuning Vision Transformers

This paper presents a parameter-efficient prompt tuning method, named PPT, to
adapt a large multi-modal model for 3D point cloud understanding. Existing
strategies are quite expensive in computation and storage, and depend on
time-consuming prompt engineering. We address the problems from three aspects.
Firstly, a PromptLearner module is devised to replace hand-crafted prompts with
learnable contexts to automate the prompt tuning process. Then, we lock the
pre-trained backbone instead of adopting the full fine-tuning paradigm to
substantially improve the parameter efficiency. Finally, a lightweight
PointAdapter module is arranged near target tasks to enhance prompt tuning for
3D point cloud understanding. Comprehensive experiments are conducted to
demonstrate the superior parameter and data efficiency of the proposed
method.Meanwhile, we obtain new records on 4 public datasets and multiple 3D
tasks, i.e., point cloud recognition, few-shot learning, and part segmentation.
The implementation is available at this https URL

该研究提出了一种参数高效的提示调优方法（称为 PPT），用于适应大型多模态模型的 3D 点云理解。通过 PromptLearner 模块和 PointAdapter 模块，在提高参数效率的同时增强了对 3D 点云理解的提示调优能力，实验证明了该方法在参数和数据效率方面的优越性。