Despite recent advancements, NLP models continue to be vulnerable to bias.
This bias often originates from the uneven distribution of real-world data and
can propagate through the annotation process. Escalated integration of these
models in our lives calls for methods to mitigate bias without overbearing
annotation costs. While active learning (AL) has shown promise in training
models with a small amount of annotated data, AL's reliance on the model's
behavior for selective sampling can lead to an accumulation of unwanted bias
rather than bias mitigation. However, infusing clustering with AL can overcome
the bias issue of both AL and traditional annotation methods while exploiting
AL's annotation efficiency. In this paper, we propose a novel adaptive
clustering-based active learning algorithm, D-CALM, that dynamically adjusts
clustering and annotation efforts in response to an estimated classifier
error-rate. Experiments on eight datasets for a diverse set of text
classification tasks, including emotion, hatespeech, dialog act, and book type
detection, demonstrate that our proposed algorithm significantly outperforms
baseline AL approaches with both pretrained transformers and traditional
Support Vector Machines. D-CALM showcases robustness against different measures
of information gain and, as evident from our analysis of label and error
distribution, can significantly reduce unwanted model bias.

本文提出了一种新颖的自适应基于聚类的主动学习算法 D-CALM，通过将聚类与主动学习相结合，动态地调整聚类和注释工作以响应分类器错误率的估计。实验证明，D-CALM 可以显著优于基线主动学习方法，具有较强的鲁棒性并可以显著降低模型的不想要偏差。