This paper explores the correlation between linguistic diversity, sentiment
analysis and transformer model architectures. We aim to investigate how
different English variations impact transformer-based models for irony
detection. To conduct our study, we used the EPIC corpus to extract five
diverse English variation-specific datasets and applied the KEN pruning
algorithm on five different architectures. Our results reveal several
similarities between optimal subnetworks, which provide insights into the
linguistic variations that share strong resemblances and those that exhibit
greater dissimilarities. We discovered that optimal subnetworks across models
share at least 60% of their parameters, emphasizing the significance of
parameter values in capturing and interpreting linguistic variations. This
study highlights the inherent structural similarities between models trained on
different variants of the same language and also the critical role of parameter
values in capturing these nuances.

本研究探讨语言多样性、情感分析和变压器模型体系结构之间的相关性，旨在调查不同的英语变体对于基于变压器模型的讽刺检测的影响。我们使用 EPIC 语料库提取了五个不同的英语变体特定数据集，并在五个不同的体系结构上应用了 KEN 修剪算法，从而进行了研究。我们的结果揭示了最佳子网络之间的几个相似之处，这些相似之处提供了对共享强烈相似之处和表现出更大差异的语言变体的见解。我们发现，跨模型的最佳子网络至少共享 60% 的参数，强调了参数值在捕捉和解释语言变化方面的重要性。本研究突显了在不同语言变体上训练的模型之间固有的结构相似性，并强调了参数值在捕捉这些细微差别中的关键作用。

Transformer 模型中的语言指纹：语言变体对讽刺检测参数选择的影响

Linguistic Fingerprint in Transformer Models: How Language Variation  Influences Parameter Selection in Irony Detection

Objective: To comparatively evaluate several transformer model architectures
at identifying protected health information (PHI) in the i2b2/UTHealth 2014
clinical text de-identification challenge corpus.
Methods: The i2b2/UTHealth 2014 corpus contains N=1304 clinical notes
obtained from N=296 patients. Using a transfer learning framework, we fine-tune
several transformer model architectures on the corpus, including: BERT-base,
BERT-large, ROBERTA-base, ROBERTA-large, ALBERT-base and ALBERT-xxlarge. During
fine-tuning we vary the following model hyper-parameters: batch size, number
training epochs, learning rate and weight decay. We fine tune models on a
training data set, we evaluate and select optimally performing models on an
independent validation dataset, and lastly assess generalization performance on
a held-out test dataset. We assess model performance in terms of accuracy,
precision (positive predictive value), recall (sensitivity) and F1 score
(harmonic mean of precision and recall). We are interested in overall model
performance (PHI identified vs. PHI not identified), as well as PHI-specific
model performance.
Results: We observe that the ROBERTA-large models perform best at identifying
PHI in the i2b2/UTHealth 2014 corpus, achieving >99% overall accuracy and 96.7%
recall/precision on the heldout test corpus. Performance was good across many
PHI classes; however, accuracy/precision/recall decreased for identification of
the following entity classes: professions, organizations, ages, and certain
locations.
Conclusions: Transformers are a promising model class/architecture for
clinical text de-identification. With minimal hyper-parameter tuning
transformers afford researchers/clinicians the opportunity to obtain (near)
state-of-the-art performance.

比较评估了几种变压器模型架构在识别医疗文本中的受保护健康信息方面的表现，ROBERTA-large 模型在 i2b2/UTHealth 2014 语料库中的表现最佳。