Cascading multiple pre-trained models is an effective way to compose an
end-to-end system. However, fine-tuning the full cascaded model is parameter
and memory inefficient and our observations reveal that only applying adapter
modules on cascaded model can not achieve considerable performance as
fine-tuning. We propose an automatic and effective adaptive learning method to
optimize end-to-end cascaded multi-task models based on Neural Architecture
Search (NAS) framework. The candidate adaptive operations on each specific
module consist of frozen, inserting an adapter and fine-tuning. We further add
a penalty item on the loss to limit the learned structure which takes the
amount of trainable parameters into account. The penalty item successfully
restrict the searched architecture and the proposed approach is able to search
similar tuning scheme with hand-craft, compressing the optimizing parameters to
8.7% corresponding to full fine-tuning on SLURP with an even better
performance.

利用神经架构搜索（NAS）框架提出一种自动且有效的自适应学习方法来优化端到端级联多任务模型，通过在每个特定模块上选择冻结、插入适配器和微调等候选自适应操作，并在损失函数中添加了一个惩罚项限制学习结构，成功地压缩优化参数至相当于全微调的 8.7% 并获得更好的性能。