Existing pre-trained language models (PLMs) have demonstrated the
effectiveness of self-supervised learning for a broad range of natural language
processing (NLP) tasks. However, most of them are not explicitly aware of
domain-specific knowledge, which is essential for downstream tasks in many
domains, such as tasks in e-commerce scenarios. In this paper, we propose
K-PLUG, a knowledge-injected pre-trained language model based on the
encoder-decoder transformer that can be transferred to both natural language
understanding and generation tasks. We verify our method in a diverse range of
e-commerce scenarios that require domain-specific knowledge. Specifically, we
propose five knowledge-aware self-supervised pre-training objectives to
formulate the learning of domain-specific knowledge, including e-commerce
domain-specific knowledge-bases, aspects of product entities, categories of
product entities, and unique selling propositions of product entities. K-PLUG
achieves new state-of-the-art results on a suite of domain-specific NLP tasks,
including product knowledge base completion, abstractive product summarization,
and multi-turn dialogue, significantly outperforms baselines across the board,
which demonstrates that the proposed method effectively learns a diverse set of
domain-specific knowledge for both language understanding and generation tasks.

本研究提出了一种基于 encoder-decoder transformer 的知识注入预训练语言模型 K-PLUG，利用五种知识感知的自我监督预训练目标，包括电子商务领域特定知识库、产品实体的方面、产品实体的类别和产品实体的独特卖点，有效地为语言理解和生成任务学习了各种领域特定知识，并在包括产品知识库完成、摘要产品总结和多次对话等领域特定 NLP 任务中达到了新的最佳性能。