Most of the existing pre-trained language representation models neglect to
consider the linguistic knowledge of texts, which can promote language
understanding in NLP tasks. To benefit the downstream tasks in sentiment
analysis, we propose a novel language representation model called SentiLARE,
which introduces word-level linguistic knowledge including part-of-speech tag
and sentiment polarity (inferred from SentiWordNet) into pre-trained models. We
first propose a context-aware sentiment attention mechanism to acquire the
sentiment polarity of each word with its part-of-speech tag by querying
SentiWordNet. Then, we devise a new pre-training task called label-aware masked
language model to construct knowledge-aware language representation.
Experiments show that SentiLARE obtains new state-of-the-art performance on a
variety of sentiment analysis tasks.

为了提高 NLP 任务中的语言理解能力，在考虑了文本的语言学知识（例如词性标注和情感极性）的基础上，提出了一种新的语言表示模型 SentiLARE，并使用标签感知掩蔽语言模型进行预训练，实验结果表明 SentiLARE 在各种情感分析任务上取得了最新的最佳性能。