Prior studies in privacy policies frame the question answering (QA) tasks as
identifying the most relevant text segment or a list of sentences from the
policy document for a user query. However, annotating such a dataset is
challenging as it requires specific domain expertise (e.g., law academics).
Even if we manage a small-scale one, a bottleneck that remains is that the
labeled data are heavily imbalanced (only a few segments are relevant)
--limiting the gain in this domain. Therefore, in this paper, we develop a
novel data augmentation framework based on ensembling retriever models that
captures the relevant text segments from unlabeled policy documents and expand
the positive examples in the training set. In addition, to improve the
diversity and quality of the augmented data, we leverage multiple pre-trained
language models (LMs) and cascaded them with noise reduction oracles. Using our
augmented data on the PrivacyQA benchmark, we elevate the existing baseline by
a large margin (10% F1) and achieve a new state-of-the-art F1 score of 50%. Our
ablation studies provide further insights into the effectiveness of our
approach.

本文通过集成检索模型和利用多个预训练语言模型及去噪防护，开发了一种新型数据增强框架，以捕获未标记的策略文件中的相关文本段，并扩展训练集中的正面示例。使用此增强数据，该研究在 PrivacyQA 基准测试中将现有水平提高了 10% F1，实现了 50% 的新水平。（其中，F1 是一种综合度量，用于评估二元分类器的准确性。）