Arabic is a Semitic language which is widely spoken with many dialects. Given
the success of pre-trained language models, many transformer models trained on
Arabic and its dialects have surfaced. While there have been an extrinsic
evaluation of these models with respect to downstream NLP tasks, no work has
been carried out to analyze and compare their internal representations. We
probe how linguistic information is encoded in the transformer models, trained
on different Arabic dialects. We perform a layer and neuron analysis on the
models using morphological tagging tasks for different dialects of Arabic and a
dialectal identification task. Our analysis enlightens interesting findings
such as: i) word morphology is learned at the lower and middle layers, ii)
while syntactic dependencies are predominantly captured at the higher layers,
iii) despite a large overlap in their vocabulary, the MSA-based models fail to
capture the nuances of Arabic dialects, iv) we found that neurons in embedding
layers are polysemous in nature, while the neurons in middle layers are
exclusive to specific properties

对在不同阿拉伯语方言上训练的 transformer 模型进行层与神经元分析，研究有趣的发现如下：①单词形态在较低和中间层次上学习，②句法依赖在较高层次上被捕获，③虽然 MSA 模型的词汇表与方言词汇表有很大重叠，但仍然无法捕捉阿拉伯语方言的微妙之处，④嵌入层的神经元是一词多义的，而中间层的神经元仅适用于特定属性。