Leveraging large amounts of unlabeled data using Transformer-like architectures, like BERT, has gained popularity in recent times owing to their effectiveness in learning general representations that can then be further fine-tuned for downstream tasks to much success. However, training these models can be costly both from an economic and environmental standpoint. In this work, we investigate how to effectively use unlabeled data: by exploring the task-specific semi-supervised approach, Cross-View Training (CVT) and comparing it with task-agnostic BERT in multiple settings that include domain and task relevant English data. CVT uses a much lighter model architecture and we show that it achieves similar performance to BERT on a set of sequence tagging tasks, with lesser financial and environmental impact.

本文探讨了如何有效地利用未标记数据——通过探索任务特定的半监督方法Cross-View Training (CVT)，并将其与任务不可知的BERT在包括领域和任务相关的英语数据的多个设置中进行比较。CVT使用较轻的模型结构，并表明它在一组序列标记任务上实现了与BERT相似的性能，具有更小的经济和环境影响。

BERT与否：比较针对任务与任务不可知的半监督序列标注方法