Even when aggregate accuracy is high, state-of-the-art NLP models often fail
systematically on specific subgroups of data, resulting in unfair outcomes and
eroding user trust. Additional data collection may not help in addressing these
weaknesses, as such challenging subgroups may be unknown to users, and
underrepresented in the existing and new data. We propose Targeted Data
Generation (TDG), a framework that automatically identifies challenging
subgroups, and generates new data for those subgroups using large language
models (LLMs) with a human in the loop. TDG estimates the expected benefit and
potential harm of data augmentation for each subgroup, and selects the ones
most likely to improve within group performance without hurting overall
performance. In our experiments, TDG significantly improves the accuracy on
challenging subgroups for state-of-the-art sentiment analysis and natural
language inference models, while also improving overall test accuracy.

为解决 NLP 模型在特定子群体上的不公平性和准确性问题，提出了一种名为 'Targeted Data Generation（TDG）' 的框架，它可以自动识别具有挑战性的子群体并使用大型语言模型（LLMs）来生成新数据，以提高模型的准确性。