Recently, many datasets have been proposed to test the systematic generalization ability of neural networks. The companion baseline Transformers, typically trained with default hyper-parameters from standard tasks, are shown to fail dramatically. Here we demonstrate that by revisiting model configurations as basic as scaling of embeddings, early stopping, relative positional embedding, and Universal Transformer variants, we can drastically improve the performance of Transformers on systematic generalization. We report improvements on five popular datasets: SCAN, CFQ, PCFG, COGS, and Mathematics dataset. Our models improve accuracy from 50% to 85% on the PCFG productivity split, and from 35% to 81% on COGS. On SCAN, relative positional embedding largely mitigates the EOS decision problem (Newman et al., 2020), yielding 100% accuracy on the length split with a cutoff at 26. Importantly, performance differences between these models are typically invisible on the IID data split. This calls for proper generalization validation sets for developing neural networks that generalize systematically. We publicly release the code to reproduce our results.

通过重新设计模型配置，如embedding、early stopping、相对位置嵌入和Universal Transformer变体等基本元素，我们可以显著提高Transformers在系统泛化方面的性能，同时在PLE、COGS、SCAN等数据集上显示性能的提高，从而表明在开发具有系统性泛化能力的神经网络时需要恰当的泛化验证集。

细节见魔鬼：简单技巧提高Transformer系统化泛化能力