Training large language models (LLMs) for external tool usage is a rapidly expanding field, with recent research focusing on generating synthetic data to address the shortage of available data. However, the absence of systematic data quality checks poses complications for properly training and testing models. To that end, we propose two approaches for assessing the reliability of data for training LLMs to use external tools. The first approach uses intuitive, human-defined correctness criteria. The second approach uses a model-driven assessment with in-context evaluation. We conduct a thorough evaluation of data quality on two popular benchmarks, followed by an extrinsic evaluation that showcases the impact of data quality on model performance. Our results demonstrate that models trained on high-quality data outperform those trained on unvalidated data, even when trained with a smaller quantity of data. These findings empirically support the significance of assessing and ensuring the reliability of training data for tool-using LLMs.

本研究针对在大型语言模型（LLMs）外部工具使用中数据短缺问题，通过提出两种评估合成数据可靠性的方法，填补了系统性数据质量检查的空白。研究结果表明，使用高质量数据训练的模型性能显著优于使用未经验证的数据，即使数据量较少，也能取得更好的效果，这验证了确保训练数据可靠性的必要性。

质量至关重要：评估用于工具使用的大型语言模型的合成数据