Prompt Tuning is a popular parameter-efficient finetuning method for
pre-trained large language models (PLMs). Recently, based on experiments with
RoBERTa, it has been suggested that Prompt Tuning activates specific neurons in
the transformer's feed-forward networks, that are highly predictive and
selective for the given task. In this paper, we study the robustness of Prompt
Tuning in relation to these "skill neurons", using RoBERTa and T5. We show that
prompts tuned for a specific task are transferable to tasks of the same type
but are not very robust to adversarial data, with higher robustness for T5 than
RoBERTa. At the same time, we replicate the existence of skill neurons in
RoBERTa and further show that skill neurons also seem to exist in T5.
Interestingly, the skill neurons of T5 determined on non-adversarial data are
also among the most predictive neurons on the adversarial data, which is not
the case for RoBERTa. We conclude that higher adversarial robustness may be
related to a model's ability to activate the relevant skill neurons on
adversarial data.

Prompt Tuning 是一种流行的参数高效微调方法，可以在预训练的大型语言模型（PLMs）中激活特定的技能神经元，并与 T5 和 RoBERTa 模型的对抗鲁棒性进行比较。我们发现，对特定任务进行微调的提示可以迁移到相同类型的任务，但对于对抗性数据的鲁棒性较低，其中 T5 的鲁棒性高于 RoBERTa。同时，我们还论证了 RoBERTa 和 T5 中存在技能神经元，并发现 T5 中在非对抗性数据中确定的技能神经元也是对对抗性数据具有预测性的，而 RoBERTa 则不然。我们得出结论，较高的对抗性鲁棒性可能与模型在对抗性数据上激活相关技能神经元的能力有关。