Deep transformer models have pushed performance on NLP tasks to new limits,
suggesting sophisticated treatment of complex linguistic inputs, such as
phrases. However, we have limited understanding of how these models handle
representation of phrases, and whether this reflects sophisticated composition
of phrase meaning like that done by humans. In this paper, we present
systematic analysis of phrasal representations in state-of-the-art pre-trained
transformers. We use tests leveraging human judgments of phrase similarity and
meaning shift, and compare results before and after control of word overlap, to
tease apart lexical effects versus composition effects. We find that phrase
representation in these models relies heavily on word content, with little
evidence of nuanced composition. We also identify variations in phrase
representation quality across models, layers, and representation types, and
make corresponding recommendations for usage of representations from these
models.

本文系统地分析了最新预训练 transformer 模型中对词组的表征方式，并使用人类对词组相似度和语义转换的判断来比较单词重叠与组成效应对因素间的影响。结果表明这些模型对词汇内容依赖很强，且未发现有像人类那样的精细组成效应。同时，本文还发现了在不同模型、层次和表征类型之间的词组表征质量的差异，并为使用这些模型的表征提出了相应的建议。

评估变形金刚中成语的表示和组合

Assessing Phrasal Representation and Composition in Transformers

Building meaningful phrase representations is challenging because phrase
meanings are not simply the sum of their constituent meanings. Lexical
composition can shift the meanings of the constituent words and introduce
implicit information. We tested a broad range of textual representations for
their capacity to address these issues. We found that as expected,
contextualized word representations perform better than static word embeddings,
more so on detecting meaning shift than in recovering implicit information, in
which their performance is still far from that of humans. Our evaluation suite,
including 5 tasks related to lexical composition effects, can serve future
research aiming to improve such representations.

本研究通过测试多种文本表示模型的性能，发现与静态词嵌入相比，上下文有关的单词表示模型对于检测含义转移等方面的应对能力更强，但在获取暗含信息方面，其表现仍远远不及人类水平。研究者设计的包含 5 个涉及词汇组合效应的任务的评估套件可用于提高此类表示模型的研究。