Data annotated by humans is a source of knowledge by describing the
peculiarities of the problem and therefore fueling the decision process of the
trained model. Unfortunately, the annotation process for subjective natural
language processing (NLP) problems like offensiveness or emotion detection is
often very expensive and time-consuming. One of the inevitable risks is to
spend some of the funds and annotator effort on annotations that do not provide
any additional knowledge about the specific task. To minimize these costs, we
propose a new model-based approach that allows the selection of tasks annotated
individually for each text in a multi-task scenario. The experiments carried
out on three datasets, dozens of NLP tasks, and thousands of annotations show
that our method allows up to 40% reduction in the number of annotations with
negligible loss of knowledge. The results also emphasize the need to collect a
diverse amount of data required to efficiently train a model, depending on the
subjectivity of the annotation task. We also focused on measuring the relation
between subjective tasks by evaluating the model in single-task and multi-task
scenarios. Moreover, for some datasets, training only on the labels predicted
by our model improved the efficiency of task selection as a self-supervised
learning regularization technique.

通过新的基于模型的方法，我们提出了一种在多任务场景中为每个文本选择逐个注释的任务的方式，从而最大程度地减少注释数量，并且几乎不损失知识，同时还强调了根据注释任务的主观性收集多样化数据以有效训练模型的需求，并通过在单任务和多任务场景中评估模型来衡量主观任务之间的关系，此外，对于某些数据集，仅依靠我们模型预测的标签进行训练，可作为自监督学习规范化技术，提高任务选择的效率。