Recent diagnostic datasets on compositional generalization, such as SCAN (Lake and Baroni, 2018) and COGS (Kim and Linzen, 2020), expose severe problems in models trained from scratch on these datasets. However, in contrast to this poor performance, state-of-the-art models trained on larger and more general datasets show better generalization ability. In this work, to reconcile this inconsistency, we conduct an empirical analysis by training Transformer models on a variety of training sets with different data factors, including dataset scale, pattern complexity, example difficulty, etc. First, we show that increased dataset complexity can lead to better generalization behavior on multiple different generalization challenges. To further understand this improvement, we show two axes of the benefit from more complex datasets: they provide more diverse examples so compositional understanding becomes more effective, and they also prevent ungeneralizable memorization of the examples due to reduced example repetition frequency. Finally, we explore how training examples of different difficulty levels influence generalization differently. On synthetic datasets, simple examples invoke stronger compositionality than hard examples do. On larger-scale real language datasets, while hard examples become more important potentially to ensure decent data coverage, a balanced mixture of simple and hard examples manages to induce the strongest generalizability. The code and data for this work are available at https://github.com/owenzx/data4comp

通过对不同数据因素（包括数据集规模、模式复杂度和示例难度等）的多样化训练集进行实证分析，本研究发现增加数据集复杂性有助于提高模型在多个不同泛化挑战上的泛化行为，并且展示了更复杂的数据集提供更多样化示例以增强组合性理解效果，并减少示例的重复频率以避免不可泛化的记忆，同时强调了简单示例在合成数据集上引发比复杂示例更强的组合性，而在更大规模的真实语言数据集上，简单和复杂示例的平衡混合能够诱导出最强的泛化能力。

更好的组合泛化数据因子