Pre-trained encoder-only and sequence-to-sequence (seq2seq) models each have
advantages, however training both model types from scratch is computationally
expensive. We explore recipes to improve pre-training efficiency by
initializing one model from the other. (1) Extracting the encoder from a
seq2seq model, we show it under-performs a Masked Language Modeling (MLM)
encoder, particularly on sequence labeling tasks. Variations of masking during
seq2seq training, reducing the decoder size, and continuing with a small amount
of MLM training do not close the gap. (2) Conversely, using an encoder to
warm-start seq2seq training, we show that by unfreezing the encoder partway
through training, we can match task performance of a from-scratch seq2seq
model. Overall, this two-stage approach is an efficient recipe to obtain both a
multilingual encoder and a seq2seq model, matching the performance of training
each model from scratch while reducing the total compute cost by 27%.

本研究通过将基于编码器和基于序列到序列模型初始化来提高预训练效率，并展示了这种双阶段方法可以获得多语言编码器和序列到序列模型的表现，同时减少了总计算成本约 27%。

多语言编码器和 Seq2Seq 模型的顺序预训练配方

Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq  Models

General purpose relation extraction has recently seen considerable gains in
part due to a massively data-intensive distant supervision technique from
Soares et al. (2019) that produces state-of-the-art results across many
benchmarks. In this work, we present a methodology for collecting high quality
training data for relation extraction from unlabeled text that achieves a
near-recreation of their zero-shot and few-shot results at a fraction of the
training cost. Our approach exploits the predictable distributional structure
of date-marked news articles to build a denoised corpus -- the extraction
process filters out low quality examples. We show that a smaller multilingual
encoder trained on this corpus performs comparably to the current
state-of-the-art (when both receive little to no fine-tuning) on few-shot and
standard relation benchmarks in English and Spanish despite using many fewer
examples (50k vs. 300mil+).

该论文提出了一种新的方法，利用大规模无标签文本中可预测的分布式数据结构从而构建了一个去噪的语料库，该方法通过远成本的监督技术以少量的样本数训练出了类似于 Soares 等人的零样本和少样本结果，该技术通过小型的多语言编码器在英语和西班牙语的一些标准关系评估中实现了与当前领先技术相当的结果。