Leveraging large amounts of unlabeled data using Transformer-like
architectures, like BERT, has gained popularity in recent times owing to their
effectiveness in learning general representations that can then be further
fine-tuned for downstream tasks to much success. However, training these models
can be costly both from an economic and environmental standpoint. In this work,
we investigate how to effectively use unlabeled data: by exploring the
task-specific semi-supervised approach, Cross-View Training (CVT) and comparing
it with task-agnostic BERT in multiple settings that include domain and task
relevant English data. CVT uses a much lighter model architecture and we show
that it achieves similar performance to BERT on a set of sequence tagging
tasks, with lesser financial and environmental impact.

本文探讨了如何有效地利用未标记数据 —— 通过探索任务特定的半监督方法 Cross-View Training (CVT)，并将其与任务不可知的 BERT 在包括领域和任务相关的英语数据的多个设置中进行比较。CVT 使用较轻的模型结构，并表明它在一组序列标记任务上实现了与 BERT 相似的性能，具有更小的经济和环境影响。

BERT 与否：比较针对任务与任务不可知的半监督序列标注方法

To BERT or Not to BERT: Comparing Task-specific and Task-agnostic  Semi-Supervised Approaches for Sequence Tagging

Unsupervised representation learning algorithms such as word2vec and ELMo
improve the accuracy of many supervised NLP models, mainly because they can
take advantage of large amounts of unlabeled text. However, the supervised
models only learn from task-specific labeled data during the main training
phase. We therefore propose Cross-View Training (CVT), a semi-supervised
learning algorithm that improves the representations of a Bi-LSTM sentence
encoder using a mix of labeled and unlabeled data. On labeled examples,
standard supervised learning is used. On unlabeled examples, CVT teaches
auxiliary prediction modules that see restricted views of the input (e.g., only
part of a sentence) to match the predictions of the full model seeing the whole
input. Since the auxiliary modules and the full model share intermediate
representations, this in turn improves the full model. Moreover, we show that
CVT is particularly effective when combined with multi-task learning. We
evaluate CVT on five sequence tagging tasks, machine translation, and
dependency parsing, achieving state-of-the-art results.

本文提出一种半监督学习算法 Cross-View Training, 结合无标签文本与有标签数据，通过学习辅助预测模型来提升双向长短时记忆网络（Bi-LSTM）的表示学习能力，取得了在五项序列标记任务，机器翻译和依存句法分析等领域的最优结果。