The Lottery Ticket Hypothesis suggests that an over-parametrized network
consists of ``lottery tickets'', and training a certain collection of them
(i.e., a subnetwork) can match the performance of the full model. In this
paper, we study such a collection of tickets, which is referred to as ``winning
tickets'', in extremely over-parametrized models, e.g., pre-trained language
models. We observe that at certain compression ratios, the generalization
performance of the winning tickets can not only match but also exceed that of
the full model. In particular, we observe a phase transition phenomenon: As the
compression ratio increases, generalization performance of the winning tickets
first improves then deteriorates after a certain threshold. We refer to the
tickets on the threshold as ``super tickets''. We further show that the phase
transition is task and model dependent -- as the model size becomes larger and
the training data set becomes smaller, the transition becomes more pronounced.
Our experiments on the GLUE benchmark show that the super tickets improve
single task fine-tuning by $0.9$ points on BERT-base and $1.0$ points on
BERT-large, in terms of task-average score. We also demonstrate that adaptively
sharing the super tickets across tasks benefits multi-task learning.

该研究探讨了 Lottery Ticket Hypothesis 理论，在预训练语言模型这样极度超参数化的模型中，如何通过压缩 ratio 选定一组 “winning tickets” 子模型来匹配全模型的性能。另外观察到当 compression ratio 达到某个临界点时，性能会出现相位转变现象，我们称此时被删减对临界点最立竿见影的超票为 “super tickets”，实验证明它们对 BERT-base 和 BERT-large 的单任务微调平均得分分别有 0.9 和 1.0 的提升，多任务学习时的共享也取得了提升。