Data selection is critical for enhancing the performance of language models, particularly when aligning training datasets with a desired target distribution. This study explores the effects of different data selection methods and feature types on model performance. We evaluate whether selecting data subsets can influence downstream tasks, whether n-gram features improve alignment with target distributions, and whether embedding-based neural features provide complementary benefits. Through comparative experiments using baseline random selection methods and distribution aligned approaches, we provide insights into the interplay between data selection strategies and model training efficacy. All code for this study can be found on \href{https://github.com/jgu13/HIR-Hybrid-Importance-Resampling-for-Language-Models}{github repository}.

本研究解决了数据选择对语言模型性能提升的关键问题，特别是如何将训练数据集与目标分布对齐。我们采用了不同的数据选择方法和特征类型进行实验，发现合理的数据子集选择及n-gram特征能有效提高模型在下游任务中的表现，并且嵌入式神经特征带来额外的收益。这些发现为理解数据选择策略与模型训练效果之间的关系提供了重要见解。

数据选择策略对语言模型性能的影响研究