Masked language models (MLMs) are pretrained with a denoising objective that, while useful, is in a mismatch with the objective of downstream fine-tuning. We propose pragmatic masking and surrogate fine-tuning as two strategies that exploit social cues to drive pre-trained representations toward a broad set of concepts useful for a wide class of social meaning tasks. To test our methods, we introduce a new benchmark of 15 different Twitter datasets for social meaning detection. Our methods achieve 2.34% F1 over a competitive baseline, while outperforming other transfer learning methods such as multi-task learning and domain-specific language models pretrained on large datasets. With only 5% of training data (severely few-shot), our methods enable an impressive 68.74% average F1, and we observe promising results in a zero-shot setting involving six datasets from three different languages.

提出 pragmatic masking 和 surrogate fine-tuning 作为两种补充策略，利用社交线索指导预训练的表示，使其朝向一组广泛有用的概念，这些概念适用于各种社交意义任务。该方法在 $15$ 个不同的 Twitter 数据集上进行测试，证明了其在社交含义检测方面的有效性，且在少样本学习方面表现出色。方法还语言无关，在零样本情况下也能够有效。

利用语用掩蔽和替代微调改进社会意义检测