Modern large-scale Pre-trained Language Models (PLMs) have achieved
tremendous success on a wide range of downstream tasks. However, most of the LM
pre-training objectives only focus on text reconstruction, but have not sought
to learn latent-level interpretable representations of sentences. In this
paper, we manage to push the language models to obtain a deeper understanding
of sentences by proposing a new pre-training objective, Sparse Latent Typing,
which enables the model to sparsely extract sentence-level keywords with
diverse latent types. Experimental results show that our model is able to learn
interpretable latent type categories in a self-supervised manner without using
any external knowledge. Besides, the language model pre-trained with such an
objective also significantly improves Information Extraction related downstream
tasks in both supervised and few-shot settings. Our code is publicly available
at: this https URL.

该研究提出一种新的预训练目标 Sparse Latent Typing，让模型能够从文本中提取具有多样化潜在类型的稀疏关键词，实现对句子的深层理解，结果表明该模型通过自监督学习能够学习到可解释的潜在类型分类，同时显著提高了有监督和少样本信息抽取任务的效果。