We investigate the efficacy of visual prompting to adapt large-scale models
in vision. Following the recent approach from prompt tuning and adversarial
reprogramming, we learn a single image perturbation such that a frozen model
prompted with this perturbation performs a new task. Through comprehensive
experiments, we demonstrate that visual prompting is particularly effective for
CLIP and robust to distribution shift, achieving performance competitive with
standard linear probes. We further analyze properties of the downstream
dataset, prompt design, and output transformation in regard to adaptation
performance. The surprising effectiveness of visual prompting provides a new
perspective on adapting pre-trained models in vision. Code is available at
this http URL .

通过视觉提示来适应视觉中的大规模模型，这种方法在适应预先训练模型方面非常有效。

探索视觉提示以适应大规模模型

Exploring Visual Prompts for Adapting Large-Scale Models

In this study, we propose a novel adversarial reprogramming (AR) approach for
low-resource spoken command recognition (SCR), and build an AR-SCR system. The
AR procedure aims to modify the acoustic signals (from the target domain) to
repurpose a pretrained SCR model (from the source domain). To solve the label
mismatches between source and target domains, and further improve the stability
of AR, we propose a novel similarity-based label mapping technique to align
classes. In addition, the transfer learning (TL) technique is combined with the
original AR process to improve the model adaptation capability. We evaluate the
proposed AR-SCR system on three low-resource SCR datasets, including Arabic,
Lithuanian, and dysarthric Mandarin speech. Experimental results show that with
a pretrained AM trained on a large-scale English dataset, the proposed AR-SCR
system outperforms the current state-of-the-art results on Arabic and
Lithuanian speech commands datasets, with only a limited amount of training
data.

该研究提出了一种新的对抗重编程（AR）方法，用于低资源语音指令识别（SCR），并构建了一个 AR-SCR 系统。实验结果表明，该方法在阿拉伯语和立陶宛语命令数据集上优于现有技术，但在仅有有限训练数据的情况下。

基于相似性映射的神经模型重编程用于低资源口语命令分类

Neural Model Reprogramming with Similarity Based Mapping for  Low-Resource Spoken Command Classification

Transfer learning from pretrained language models recently became the
dominant approach for solving many NLP tasks. A common approach to transfer
learning for multiple tasks that maximize parameter sharing trains one or more
task-specific layers on top of the language model. In this paper, we present an
alternative approach based on adversarial reprogramming, which extends earlier
work on automatic prompt generation. Adversarial reprogramming attempts to
learn task-specific word embeddings that, when concatenated to the input text,
instruct the language model to solve the specified task. Using up to 25K
trainable parameters per task, this approach outperforms all existing methods
with up to 25M trainable parameters on the public leaderboard of the GLUE
benchmark. Our method, initialized with task-specific human-readable prompts,
also works in a few-shot setting, outperforming GPT-3 on two SuperGLUE tasks
with just 32 training samples.

该研究介绍了一种基于敌对重构的迁移学习方法，使用少量的可训参数，包括任务特定的单词嵌入向量，可有效地解决自然语言处理任务并优于现有的方法，同时在少样本情况下也表现出色。

WARP: 词级对抗性重新编程

WARP: Word-level Adversarial ReProgramming

Adversarial Reprogramming has demonstrated success in utilizing pre-trained
neural network classifiers for alternative classification tasks without
modification to the original network. An adversary in such an attack scenario
trains an additive contribution to the inputs to repurpose the neural network
for the new classification task. While this reprogramming approach works for
neural networks with a continuous input space such as that of images, it is not
directly applicable to neural networks trained for tasks such as text
classification, where the input space is discrete. Repurposing such
classification networks would require the attacker to learn an adversarial
program that maps inputs from one discrete space to the other. In this work, we
introduce a context-based vocabulary remapping model to reprogram neural
networks trained on a specific sequence classification task, for a new sequence
classification task desired by the adversary. We propose training procedures
for this adversarial program in both white-box and black-box settings. We
demonstrate the application of our model by adversarially repurposing various
text-classification models including LSTM, bi-directional LSTM and CNN for
alternate classification tasks.

本论文介绍一种基于上下文的词汇重映射模型，同时提出了针对该模型的白盒和黑盒训练流程，以实现通过敌对程序将 LSTM 等文本分类模型重新用于其他分类任务。