Pre-trained transformer language models have shown remarkable performance on
a variety of NLP tasks. However, recent research has suggested that
phrase-level representations in these models reflect heavy influences of
lexical content, but lack evidence of sophisticated, compositional phrase
information. Here we investigate the impact of fine-tuning on the capacity of
contextualized embeddings to capture phrase meaning information beyond lexical
content. Specifically, we fine-tune models on an adversarial paraphrase
classification task with high lexical overlap, and on a sentiment
classification task. After fine-tuning, we analyze phrasal representations in
controlled settings following prior work. We find that fine-tuning largely
fails to benefit compositionality in these representations, though training on
sentiment yields a small, localized benefit for certain models. In follow-up
analyses, we identify confounding cues in the paraphrase dataset that may
explain the lack of composition benefits from that task, and we discuss
potential factors underlying the localized benefits from sentiment training.

研究发现，预训练的转换语言模型在很多 NLP 任务上表现出色；然而，这些模型在短语级别的表示中，除了词汇内容外，缺乏复杂的组合短语信息，进一步的 fine-tuning 只能在情感任务中局部提高性能，而在重新释义任务中则由于数据集中可能存在干扰信号的原因而不能提供改进。