It is a common belief that training deep transformers from scratch requires
large datasets. Consequently, for small datasets, people usually use shallow
and simple additional layers on top of pre-trained models during fine-tuning.
This work shows that this does not always need to be the case: with proper
initialization and optimization, the benefits of very deep transformers can
carry over to challenging tasks with small datasets, including Text-to-SQL
semantic parsing and logical reading comprehension. In particular, we
successfully train $48$ layers of transformers, comprising $24$ fine-tuned
layers from pre-trained RoBERTa and $24$ relation-aware layers trained from
scratch. With fewer training steps and no task-specific pre-training, we obtain
the state-of-the-art performance on the challenging cross-domain Text-to-SQL
parsing benchmark Spider. We achieve this by deriving a novel Data-dependent
Transformer Fixed-update initialization scheme (DT-Fixup), inspired by the
prior T-Fixup work. Further error analysis shows that increasing depth can help
improve generalization on small datasets for hard cases that require reasoning
and structural understanding.

本文指出对于少量数据集的挑战性任务，如文本到 SQL 语义解析和逻辑阅读理解等，使用预训练模型进行微调是不必要的。通过新型数据依赖 Transformer 固定更新初始化方案（DT-Fixup），我们成功训练了由 48 个 transformers 层组成的模型，在无具体预训练的情况下，只需少量的训练步骤，就可以在挑战性的交叉领域文本到 SQL 解析基准 Spider 上实现最先进的性能。