Large Transformer-based models were shown to be reducible to a smaller number
of self-attention heads and layers. We consider this phenomenon from the
perspective of the lottery ticket hypothesis, using both structured and
magnitude pruning. For fine-tuned BERT, we show that (a) it is possible to find
subnetworks achieving performance that is comparable with that of the full
model, and (b) similarly-sized subnetworks sampled from the rest of the model
perform worse. Strikingly, with structured pruning even the worst possible
subnetworks remain highly trainable, indicating that most pre-trained BERT
weights are potentially useful. We also study the "good" subnetworks to see if
their success can be attributed to superior linguistic knowledge, but find them
unstable, and not explained by meaningful self-attention patterns.

本文从彩票猜想的角度探究了大型基于 Transformer 的模型可以通过剪枝获得可比拟完整模型性能的子网络。使用结构剪枝和数量剪枝对 Fine-tuned BERT 进行实验，发现即使是最差的子网络也表现良好，表明预训练 BERT 的大多数权重可能都是有用的。

当 BERT 玩彩票时，每张彩票都是获胜的

When BERT Plays the Lottery, All Tickets Are Winning

Parallel deep learning architectures like fine-tuned BERT and MT-DNN, have
quickly become the state of the art, bypassing previous deep and shallow
learning methods by a large margin. More recently, pre-trained models from
large related datasets have been able to perform well on many downstream tasks
by just fine-tuning on domain-specific datasets . However, using powerful
models on non-trivial tasks, such as ranking and large document classification,
still remains a challenge due to input size limitations of parallel
architecture and extremely small datasets (insufficient for fine-tuning). In
this work, we introduce an end-to-end system, trained in a multi-task setting,
to filter and re-rank answers in the medical domain. We use task-specific
pre-trained models as deep feature extractors. Our model achieves the highest
Spearman's Rho and Mean Reciprocal Rank of 0.338 and 0.9622 respectively, on
the ACL-BioNLP workshop MediQA Question Answering shared-task.

本文介绍了一种在多任务设置中训练的端对端系统，使用任务特定的预训练模型作为深度特征提取器，以过滤和重新排列医学领域答案的方法来解决排名和大型文档分类等任务中输入大小限制和数据不足的问题，并在 ACL-BioNLP 研讨会 MediQA 问答共享任务上取得了 Spearman's Rho 为 0.338 和 Mean Reciprocal Rank 为 0.9622 的最高分数。