Data augmentation techniques apply transformations to existing texts to
generate additional data. The transformations may produce low-quality texts,
where the meaning of the text is changed and the text may even be mangled
beyond human comprehension. Analyzing the synthetically generated texts and
their corresponding labels is slow and demanding. To winnow out texts with
incorrect labels, we develop INSPECTOR, a human-in-the-loop data inspection
technique. INSPECTOR combines the strengths of provenance tracking techniques
with assistive labeling. INSPECTOR allows users to group related texts by their
transformation provenance, i.e., the transformations applied to the original
text, or feature provenance, the linguistic features of the original text. For
assistive labeling, INSPECTOR computes metrics that approximate data quality,
and allows users to compare the corresponding label of each text against the
predictions of a large language model. In a user study, INSPECTOR increases the
number of texts with correct labels identified by 3X on a sentiment analysis
task and by 4X on a hate speech detection task. The participants found grouping
the synthetically generated texts by their common transformation to be the most
useful technique. Surprisingly, grouping texts by common linguistic features
was perceived to be unhelpful. Contrary to prior work, our study finds that no
single technique obviates the need for human inspection effort. This validates
the design of INSPECTOR which combines both analysis of data provenance and
assistive labeling to reduce human inspection effort.

使用数据增强技术生成的合成文本及其相应的标签分析是缓慢且繁重的任务。为了消除具有错误标签的文本，我们开发了一种名为 INSPECTOR 的人机交互数据检查技术，结合了数据溯源技术与辅助标注的优势。INSPECTOR 允许用户根据原始文本的转换溯源或语言特征溯源将相关文本分组，并计算近似数据质量的指标，将每个文本的标签与大型语言模型的预测进行比较。在用户研究中，使用 INSPECTOR 的情感分析任务中，能够将具有正确标签的文本数量提高 3 倍，在仇恨言论检测任务中，能够提高 4 倍。参与者认为将合成文本按其共同的转换分组是最有用的技术，而按共同语言特征分组则认为无助于检查工作。与先前研究相反，我们的研究发现没有一种单一技术能够完全取代人工检查工作的需要，这验证了 INSPECTOR 设计将数据溯源分析和辅助标注相结合以减少人工检查工作的目标。