How does the importance of positional encoding in pre-trained language models
(PLMs) vary across languages with different morphological complexity? In this
paper, we offer the first study addressing this question, encompassing 23
morphologically diverse languages and 5 different downstream tasks. We choose
two categories of tasks: syntactic tasks (part-of-speech tagging, named entity
recognition, dependency parsing) and semantic tasks (natural language
inference, paraphrasing). We consider language-specific BERT models trained on
monolingual corpus for our investigation. The main experiment consists of
nullifying the effect of positional encoding during fine-tuning and
investigating its impact across various tasks and languages. Our findings
demonstrate that the significance of positional encoding diminishes as the
morphological complexity of a language increases. Across all experiments, we
observe clustering of languages according to their morphological typology -
with analytic languages at one end and synthetic languages at the opposite end.

本研究首次探讨了不同形态结构复杂性的语言中，预训练语言模型（PLMs）中位置编码的重要性如何变化，并囊括了 23 种形态多样的语言和 5 种不同的下游任务。通过选择句法任务和语义任务，我们从各自语言的 BERT 模型中进行研究，实证发现位置编码的显著性随着语言形态复杂性的增加而降低，且观察到语言根据其形态结构的分类呈现聚类现象，其中分析型语言位于一端，而综合型语言位于另一端。

基于形态学的位置编码研究

A Morphology-Based Investigation of Positional Encodings

Large Language Models (LLMs) exhibit impressive performance on a range of NLP
tasks, due to the general-purpose linguistic knowledge acquired during
pretraining. Existing model interpretability research (Tenney et al., 2019)
suggests that a linguistic hierarchy emerges in the LLM layers, with lower
layers better suited to solving syntactic tasks and higher layers employed for
semantic processing. Yet, little is known about how encodings of different
linguistic phenomena interact within the models and to what extent processing
of linguistically-related categories relies on the same, shared model
representations. In this paper, we propose a framework for testing the joint
encoding of linguistic categories in LLMs. Focusing on syntax, we find evidence
of joint encoding both at the same (related part-of-speech (POS) classes) and
different (POS classes and related syntactic dependency relations) levels of
linguistic hierarchy. Our cross-lingual experiments show that the same patterns
hold across languages in multilingual LLMs.

大型语言模型研究中发现的语言层次结构及其对语法任务的编码方式提供可解释性证据。

探究 LLMs 对语言类别的联合编码

Probing LLMs for Joint Encoding of Linguistic Categories

Contextualized word representations, such as ELMo and BERT, were shown to
perform well on various semantic and syntactic tasks. In this work, we tackle
the task of unsupervised disentanglement between semantics and structure in
neural language representations: we aim to learn a transformation of the
contextualized vectors, that discards the lexical semantics, but keeps the
structural information. To this end, we automatically generate groups of
sentences which are structurally similar but semantically different, and use
metric-learning approach to learn a transformation that emphasizes the
structural component that is encoded in the vectors. We demonstrate that our
transformation clusters vectors in space by structural properties, rather than
by lexical semantics. Finally, we demonstrate the utility of our distilled
representations by showing that they outperform the original contextualized
representations in a few-shot parsing setting.

本文讲述了如何通过自动生成语义不同但结果相似的句子组集来学习一种转换，以去除词法语义但保留结构信息，并证明这种方法在结构聚类方面的效果优于词汇语义聚类方法，最终在少样本解析任务中优于原始上下文化语言表示。

从上下文化单词表示中无监督地提取句法信息

Unsupervised Distillation of Syntactic Information from Contextualized  Word Representations

At present, different deep learning models are presenting high accuracy on
popular inference datasets such as SNLI, MNLI, and SciTail. However, there are
different indicators that those datasets can be exploited by using some simple
linguistic patterns. This fact poses difficulties to our understanding of the
actual capacity of machine learning models to solve the complex task of textual
inference. We propose a new set of syntactic tasks focused on contradiction
detection that require specific capacities over linguistic logical forms such
as: Boolean coordination, quantifiers, definite description, and counting
operators. We evaluate two kinds of deep learning models that implicitly
exploit language structure: recurrent models and the Transformer network BERT.
We show that although BERT is clearly more efficient to generalize over most
logical forms, there is space for improvement when dealing with counting
operators. Since the syntactic tasks can be implemented in different languages,
we show a successful case of cross-lingual transfer learning between English
and Portuguese.

本文提出了一种新的语法任务集，专注于矛盾检测，并评估了递归模型和 BERT 网络。虽然 BERT 在大多数逻辑形式上具有更好的泛化效果，但在处理计数算子时仍有待改进，同时表明该语法任务可以在不同的语言中实现并实现跨语言传递学习。