Deep transformer models have pushed performance on NLP tasks to new limits,
suggesting sophisticated treatment of complex linguistic inputs, such as
phrases. However, we have limited understanding of how these models handle
representation of phrases, and whether this reflects sophisticated composition
of phrase meaning like that done by humans. In this paper, we present
systematic analysis of phrasal representations in state-of-the-art pre-trained
transformers. We use tests leveraging human judgments of phrase similarity and
meaning shift, and compare results before and after control of word overlap, to
tease apart lexical effects versus composition effects. We find that phrase
representation in these models relies heavily on word content, with little
evidence of nuanced composition. We also identify variations in phrase
representation quality across models, layers, and representation types, and
make corresponding recommendations for usage of representations from these
models.

本文系统地分析了最新预训练 transformer 模型中对词组的表征方式，并使用人类对词组相似度和语义转换的判断来比较单词重叠与组成效应对因素间的影响。结果表明这些模型对词汇内容依赖很强，且未发现有像人类那样的精细组成效应。同时，本文还发现了在不同模型、层次和表征类型之间的词组表征质量的差异，并为使用这些模型的表征提出了相应的建议。