Parameter efficient adaptation methods have become a key mechanism to train
large pre-trained models for downstream tasks. However, their per-task
parameter overhead is considered still high when the number of downstream tasks
to adapt for is large. We introduce an adapter module that has a better
efficiency in large scale multi-task adaptation scenario. Our adapter is
hierarchical in terms of how the adapter parameters are allocated. The adapter
consists of a single shared controller network and multiple task-level adapter
heads to reduce the per-task parameter overhead without performance regression
on downstream tasks. The adapter is also recurrent so the entire adapter
parameters are reused across different layers of the pre-trained model. Our
Hierarchical Recurrent Adapter (HRA) outperforms the previous adapter-based
approaches as well as full model fine-tuning baseline in both single and
multi-task adaptation settings when evaluated on automatic speech recognition
tasks.

参数高效的适应方法在训练大型预训练模型进行下游任务时已成为关键机制，我们引入一种适应器模块，它在大规模多任务适应情景中具有更好的效率，适应器具有层次分配适应器参数的特点，通过单个共享的控制器网络和多个任务级适应器头减少每个任务的参数开销，同时避免对下游任务的性能衰退，适应器也是循环的，因此整个适应器参数在预训练模型的不同层之间被重复使用，我们的分层循环适应器 (HRA) 在自动语音识别任务上在单任务和多任务适应设置中表现优于之前的基于适应器的方法和完整模型微调基准线。