Pre-trained language models such as BERT have achieved great success in a
broad range of natural language processing tasks. However, BERT cannot well
support E-commerce related tasks due to the lack of two levels of domain
knowledge, i.e., phrase-level and product-level. On one hand, many E-commerce
tasks require an accurate understanding of domain phrases, whereas such
fine-grained phrase-level knowledge is not explicitly modeled by BERT's
training objective. On the other hand, product-level knowledge like product
associations can enhance the language modeling of E-commerce, but they are not
factual knowledge thus using them indiscriminately may introduce noise. To
tackle the problem, we propose a unified pre-training framework, namely,
E-BERT. Specifically, to preserve phrase-level knowledge, we introduce Adaptive
Hybrid Masking, which allows the model to adaptively switch from learning
preliminary word knowledge to learning complex phrases, based on the fitting
progress of two modes. To utilize product-level knowledge, we introduce
Neighbor Product Reconstruction, which trains E-BERT to predict a product's
associated neighbors with a denoising cross attention layer. Our investigation
reveals promising results in four downstream tasks, i.e., review-based question
answering, aspect extraction, aspect sentiment classification, and product
classification.

提出了一个名为 E-BERT 的统一的预训练框架，旨在解决 BERT 在 E-commerce 相关任务中无法很好支持两种级别的领域知识的问题，该框架通过自适应混合掩蔽实现了短语级别知识的保留以及通过邻居产品重构实现了利用产品级别知识，研究表明 E-BERT 在评论问答、方面提取、方面情感分类和产品分类等四个下游任务中有着良好的表现。