The pre-training for language models captures general language understanding
but fails to distinguish the affective impact of a particular context to a
specific word. Recent works have sought to introduce contrastive learning (CL)
for sentiment-aware pre-training in acquiring affective information.
Nevertheless, these methods present two significant limitations. First, the
compatibility of the GPU memory often limits the number of negative samples,
hindering the opportunities to learn good representations. In addition, using
only a few sentiment polarities as hard labels, e.g., positive, neutral, and
negative, to supervise CL will force all representations to converge to a few
points, leading to the issue of latent space collapse. This study proposes a
soft momentum contrastive learning (SoftMCL) for fine-grained sentiment-aware
pre-training. Instead of hard labels, we introduce valence ratings as
soft-label supervision for CL to fine-grained measure the sentiment
similarities between samples. The proposed SoftMCL is conducted on both the
word- and sentence-level to enhance the model's ability to learn affective
information. A momentum queue was introduced to expand the contrastive samples,
allowing storing and involving more negatives to overcome the limitations of
hardware platforms. Extensive experiments were conducted on four different
sentiment-related tasks, which demonstrates the effectiveness of the proposed
SoftMCL method. The code and data of the proposed SoftMCL is available at:
this https URL

通过软标签监督，提出了一种软动量对比学习方法（SoftMCL），用于细粒度的情感感知预训练，在单词和句子级别上增强模型学习情感信息的能力，并通过克服硬件平台的限制，扩展了对比样本的数量。实验结果表明所提出的 SoftMCL 方法的有效性。