Training Memory-based transformers can require a large amount of memory and
can be quite inefficient. We propose a novel two-phase training mechanism and a
novel regularization technique to improve the training efficiency of
memory-based transformers, which are often used for long-range context
problems. For our experiments, we consider transformer-XL as our baseline model
which is one of memorybased transformer models. We show that our resultant
model, Skip Cross-head TransformerXL, outperforms the baseline on character
level language modeling task with similar parameters and outperforms the
baseline on word level language modelling task with almost 20% fewer
parameters. Our proposed methods do not require any additional memory. We also
demonstrate the effectiveness of our regularization mechanism on BERT which
shows similar performance with reduction in standard deviation of scores of
around 30% on multiple GLUE tasks.

我们提出了一种新颖的两阶段训练机制和一种新颖的正则化技术，以提高基于内存的 Transformer 的训练效率，该模型通常用于解决长程背景问题。我们在字符级语言模型任务上展示了我们的结果模型 Skip Cross-head TransformerXL，在相似参数下优于基线模型，并在词级语言建模任务上优于基线模型，参数减少近 20%。我们提出的方法不需要额外的内存。我们还展示了我们在 BERT 上的正则化机制的有效性，该机制在多个 GLUE 任务中表现出相似性能，并降低了标准差约 30%。