Language model debiasing has emerged as an important field of study in the
NLP community. Numerous debiasing techniques were proposed, but bias ablation
remains an unaddressed issue. We demonstrate a novel framework for inspecting
bias in pre-trained transformer-based language models via movement pruning.
Given a model and a debiasing objective, our framework finds a subset of the
model containing less bias than the original model. We implement our framework
by pruning the model while fine-tuning it on the debiasing objective. Optimized
are only the pruning scores - parameters coupled with the model's weights that
act as gates. We experiment with pruning attention heads, an important building
block of transformers: we prune square blocks, as well as establish a new way
of pruning the entire heads. Lastly, we demonstrate the usage of our framework
using gender bias, and based on our findings, we propose an improvement to an
existing debiasing method. Additionally, we re-discover a bias-performance
trade-off: the better the model performs, the more bias it contains.

本研究提出了一种通过运动剪枝实现检查预训练变压器语言模型中偏差的方法，并使用性别偏见作为案例来验证方法。

探究使用运动修剪在预训练基于 Transformer 的语言模型中的性别偏见

Gender Biases and Where to Find Them: Exploring Gender Bias in Pre-Trained Transformer-based Language Models Using Movement Pruning

Active learning has been shown to be an effective way to alleviate some of
the effort required in utilising large collections of unlabelled data for
machine learning tasks without needing to fully label them. The representation
mechanism used to represent text documents when performing active learning,
however, has a significant influence on how effective the process will be.
While simple vector representations such as bag-of-words and embedding-based
representations based on techniques such as word2vec have been shown to be an
effective way to represent documents during active learning, the emergence of
representation mechanisms based on the pre-trained transformer-based neural
network models popular in natural language processing research (e.g. BERT)
offer a promising, and as yet not fully explored, alternative. This paper
describes a comprehensive evaluation of the effectiveness of representations
based on pre-trained transformer-based language models for active learning.
This evaluation shows that transformer-based models, especially BERT-like
models, that have not yet been widely used in active learning, achieve a
significant improvement over more commonly used vector representations like
bag-of-words or other classical word embeddings like word2vec. This paper also
investigates the effectiveness of representations based on variants of BERT
such as Roberta, Albert as well as comparing the effectiveness of the [CLS]
token representation and the aggregated representation that can be generated
using BERT-like models. Finally, we propose an approach Adaptive Tuning Active
Learning. Our experiments show that the limited label information acquired in
active learning can not only be used for training a classifier but can also
adaptively improve the embeddings generated by the BERT-like language models as
well.

本文全面评估了基于预训练变压器语言模型的表示法在主动学习中的有效性，发现 BERT 般的模型比 word2vec 和包 - of-words 等更常见的表示法表现更好，同时提出了一种自适应调整主动学习方法，利用有限的标签信息训练分类器和适应性改进 BERT 模型生成的嵌入。

研究基于预训练 Transformer 语言模型的表征在标记文本数据的主动学习中的有效性

Investigating the Effectiveness of Representations Based on Pretrained  Transformer-based Language Models in Active Learning for Labelling Text  Datasets