Probing has become a go-to methodology for interpreting and analyzing deep
neural models in natural language processing. However, there is still a lack of
understanding of the limitations and weaknesses of various types of probes. In
this work, we suggest a strategy for input-level intervention on naturalistic
sentences. Using our approach, we intervene on the morpho-syntactic features of
a sentence, while keeping the rest of the sentence unchanged. Such an
intervention allows us to causally probe pre-trained models. We apply our
naturalistic causal probing framework to analyze the effects of grammatical
gender and number on contextualized representations extracted from three
pre-trained models in Spanish: the multilingual versions of BERT, RoBERTa, and
GPT-2. Our experiments suggest that naturalistic interventions lead to stable
estimates of the causal effects of various linguistic properties. Moreover, our
experiments demonstrate the importance of naturalistic causal probing when
analyzing pre-trained models.

本文提出了一种自然语言中以输入层面干预的方法，使用这种方法可以对神经网络模型进行因果探究，实验结果表明，这种自然干预的方法可以稳定地估算出各种语言属性的因果影响，这在分析预训练模型时非常重要。

自然语言下形态 - 句法因果分析

Naturalistic Causal Probing for Morpho-Syntax

We investigate contract element extraction. We show that LSTM-based encoders
perform better than dilated CNNs, Transformers, and BERT in this task. We also
find that domain-specific WORD2VEC embeddings outperform generic pre-trained
GLOVE embeddings. Morpho-syntactic features in the form of POS tag and token
shape embeddings, as well as context-aware ELMO embeddings do not improve
performance. Several of these observations contradict choices or findings of
previous work on contract element extraction and generic sequence labeling
tasks, indicating that contract element extraction requires careful
task-specific choices. Analyzing the results of (i) plain TRANSFORMER-based and
(ii) BERT-based models, we find that in the examined task, where the entities
are highly context-sensitive, the lack of recurrency in TRANSFORMERs greatly
affects their performance.

本文研究了合同元素提取，表明在这一任务中，基于 LSTM 的编码器比扩张卷积神经网络、Transformer 和 BERT 表现更好；领域特定的 WORD2VEC 嵌入优于通用预训练的 GLOVE 嵌入；形态句法特征和上下文感知的 ELMO 嵌入无法提高性能。这些观察结果与先前的合同元素提取和通用序列标注任务的研究结果相矛盾，表明合同元素提取需要仔细进行任务特定的选择。分析（i）基于标准 TRANSFORMER 的模型和（ii）基于 BERT 的模型的结果，我们发现在这一任务中，实体高度依赖于上下文，而标准 TRANSFORMER 中缺乏的循环结构显著影响了它们的性能。

神经合同要素提取再审视：来自芝麻街的信件

Neural Contract Element Extraction Revisited: Letters from Sesame Street

Predicting context-dependent and non-literal utterances like sarcastic and
ironic expressions still remains a challenging task in NLP, as it goes beyond
linguistic patterns, encompassing common sense and shared knowledge as crucial
components. To capture complex morpho-syntactic features that can usually serve
as indicators for irony or sarcasm across dynamic contexts, we propose a model
that uses character-level vector representations of words, based on ELMo. We
test our model on 7 different datasets derived from 3 different data sources,
providing state-of-the-art performance in 6 of them, and otherwise offering
competitive results.

使用机器学习和基于 ELMO 的字向量表征，可以在 3 个数据源派生的 7 个数据集上提供最先进的性能，以准确预测上下文相关和非文字化表述，如讽刺和嘲讽表达式。