A significant challenge to training accurate deep learning models on privacy
policies is the cost and difficulty of obtaining a large and comprehensive set
of training data. To address these challenges, we present Calpric , which
combines automatic text selection and segmentation, active learning and the use
of crowdsourced annotators to generate a large, balanced training set for
privacy policies at low cost. Automated text selection and segmentation
simplifies the labeling task, enabling untrained annotators from crowdsourcing
platforms, like Amazon's Mechanical Turk, to be competitive with trained
annotators, such as law students, and also reduces inter-annotator agreement,
which decreases labeling cost. Having reliable labels for training enables the
use of active learning, which uses fewer training samples to efficiently cover
the input space, further reducing cost and improving class and data category
balance in the data set. The combination of these techniques allows Calpric to
produce models that are accurate over a wider range of data categories, and
provide more detailed, fine-grain labels than previous work. Our crowdsourcing
process enables Calpric to attain reliable labeled data at a cost of roughly
$0.92-$1.71 per labeled text segment. Calpric 's training process also
generates a labeled data set of 16K privacy policy text segments across 9 Data
categories with balanced positive and negative samples.

Calpric 使用自动文本选择和分割、主动学习和众包注释器相结合的方法，以低成本生成大规模、均衡的隐私政策训练集，使得不具备专业训练的注释者能够与法律学生等经过训练的注释者相竞争，降低注释成本，并通过少量训练样本高效覆盖输入空间，进一步降低成本并提高数据集的类别平衡和数据类别详细度。众包过程使得 Calpric 以大约 0.92-1.71 美元每个标注文本段的成本获得可信赖的标注数据集，其中包含了 16K 个隐私政策文本段，涵盖了 9 个数据类别，并具有平衡的正负样本。