Pre-trained language models (PLMs) like BERT have made great progress in NLP.
News articles usually contain rich textual information, and PLMs have the
potentials to enhance news text modeling for various intelligent news
applications like news recommendation and retrieval. However, most existing
PLMs are in huge size with hundreds of millions of parameters. Many online news
applications need to serve millions of users with low latency tolerance, which
poses huge challenges to incorporating PLMs in these scenarios. Knowledge
distillation techniques can compress a large PLM into a much smaller one and
meanwhile keeps good performance. However, existing language models are
pre-trained and distilled on general corpus like Wikipedia, which has some gaps
with the news domain and may be suboptimal for news intelligence. In this
paper, we propose NewsBERT, which can distill PLMs for efficient and effective
news intelligence. In our approach, we design a teacher-student joint learning
and distillation framework to collaboratively learn both teacher and student
models, where the student model can learn from the learning experience of the
teacher model. In addition, we propose a momentum distillation method by
incorporating the gradients of teacher model into the update of student model
to better transfer useful knowledge learned by the teacher model. Extensive
experiments on two real-world datasets with three tasks show that NewsBERT can
effectively improve the model performance in various intelligent news
applications with much smaller models.

本文提出了一种名为 NewsBERT 的基于预训练语言模型的知识蒸馏方法，该方法利用知识蒸馏技术将大型 PLM 压缩为更小的模型，以提高新闻智能应用的性能，并且在两个真实世界的数据集上进行了广泛的实验。