Active learning has been shown to be an effective way to alleviate some of
the effort required in utilising large collections of unlabelled data for
machine learning tasks without needing to fully label them. The representation
mechanism used to represent text documents when performing active learning,
however, has a significant influence on how effective the process will be.
While simple vector representations such as bag-of-words and embedding-based
representations based on techniques such as word2vec have been shown to be an
effective way to represent documents during active learning, the emergence of
representation mechanisms based on the pre-trained transformer-based neural
network models popular in natural language processing research (e.g. BERT)
offer a promising, and as yet not fully explored, alternative. This paper
describes a comprehensive evaluation of the effectiveness of representations
based on pre-trained transformer-based language models for active learning.
This evaluation shows that transformer-based models, especially BERT-like
models, that have not yet been widely used in active learning, achieve a
significant improvement over more commonly used vector representations like
bag-of-words or other classical word embeddings like word2vec. This paper also
investigates the effectiveness of representations based on variants of BERT
such as Roberta, Albert as well as comparing the effectiveness of the [CLS]
token representation and the aggregated representation that can be generated
using BERT-like models. Finally, we propose an approach Adaptive Tuning Active
Learning. Our experiments show that the limited label information acquired in
active learning can not only be used for training a classifier but can also
adaptively improve the embeddings generated by the BERT-like language models as
well.

本文全面评估了基于预训练变压器语言模型的表示法在主动学习中的有效性，发现 BERT 般的模型比 word2vec 和包 - of-words 等更常见的表示法表现更好，同时提出了一种自适应调整主动学习方法，利用有限的标签信息训练分类器和适应性改进 BERT 模型生成的嵌入。

研究基于预训练 Transformer 语言模型的表征在标记文本数据的主动学习中的有效性

Investigating the Effectiveness of Representations Based on Pretrained  Transformer-based Language Models in Active Learning for Labelling Text  Datasets