Pre-trained language representation models achieve remarkable state of the art across a wide range of tasks in natural language processing. One of the latest advancements is BERT, a deep pre-trained transformer that yields much better results than its predecessors do. Despite its burgeoning popularity, however, BERT has not yet been applied to document classification. This task deserves attention, since it contains a few nuances: first, modeling syntactic structure matters less for document classification than for other problems, such as natural language inference and sentiment classification. Second, documents often have multiple labels across dozens of classes, which is uncharacteristic of the tasks that BERT explores. In this paper, we describe fine-tuning BERT for document classification. We are the first to demonstrate the success of BERT on this task, achieving state of the art across four popular datasets.

本研究使用BERT进行文档分类，并展示出其在四个流行数据集上的最新成果。为了解决BERT推理的计算开销，研究者提出使用知识蒸馏技术，将BERT的知识转移到小型双向LSTM中，并使用30倍更少的参数达到了BERT-base相当的性能。该研究的主要贡献在于提高了基线水平，为未来的工作提供了基础。

DocBERT：文件分类的BERT模型