Multiword expressions (MWEs) are composed of multiple words and exhibit
variable degrees of compositionality. As such, their meanings are notoriously
difficult to model, and it is unclear to what extent this issue affects
transformer architectures. Addressing this gap, we provide the first in-depth
survey of MWE processing with transformer models. We overall find that they
capture MWE semantics inconsistently, as shown by reliance on surface patterns
and memorized information. MWE meaning is also strongly localized,
predominantly in early layers of the architecture. Representations benefit from
specific linguistic properties, such as lower semantic idiosyncrasy and
ambiguity of target expressions. Our findings overall question the ability of
transformer models to robustly capture fine-grained semantics. Furthermore, we
highlight the need for more directly comparable evaluation setups.

通过对多词表达式的深入调查，我们发现 Transformer 模型在捕捉多词表达式的语义上存在不一致性，主要依赖表面模式和记忆信息，并且其表示在架构的早期层中主要存在。另外，我们强调了更直接可比较的评估设置的需求。

基于 Transformer 模型的多词表达语义研究：一项调查

Semantics of Multiword Expressions in Transformer-Based Models: A Survey

Correctly identifying multiword expressions (MWEs) is an important task for
most natural language processing systems since their misidentification can
result in ambiguity and misunderstanding of the underlying text. In this work,
we evaluate the performance of the mBERT model for MWE identification in a
multilingual context by training it on all 14 languages available in version
1.2 of the PARSEME corpus. We also incorporate lateral inhibition and language
adversarial training into our methodology to create language-independent
embeddings and improve its capabilities in identifying multiword expressions.
The evaluation of our models shows that the approach employed in this work
achieves better results compared to the best system of the PARSEME 1.2
competition, MTLB-STRUCT, on 11 out of 14 languages for global MWE
identification and on 12 out of 14 languages for unseen MWE identification.
Additionally, averaged across all languages, our best approach outperforms the
MTLB-STRUCT system by 1.23% on global MWE identification and by 4.73% on unseen
global MWE identification.

为在多语境环境下准确识别多词表达式 (MWEs)，在所有第 1.2 版 PARSEME 语料库中可用的 14 种语言上对 mBERT 模型进行训练，并将侧向抑制和语言对抗性训练纳入方法中，以创建语言无关的嵌入并提高其识别多词表达式的能力，该方法在全球 MWE 识别中比 PARSEME 1.2 竞赛的最佳系统 MTLB-STRUCT 在 11 种语言中更好地实现，对于未见 MWE 识别中的 12 种语言，平均所有语言，我们的最佳方法在全球 MWE 识别上优于 MTLB-STRUCT 系统 1.23％，在未见全球 MWE 识别上优于 MTLB-STRUCT 系统 4.73％。