Arabic is a Semitic language which is widely spoken with many dialects. Given
the success of pre-trained language models, many transformer models trained on
Arabic and its dialects have surfaced. While these models have been compared
with respect to downstream NLP tasks, no evaluation has been carried out to
directly compare the internal representations. We probe how linguistic
information is encoded in Arabic pretrained models, trained on different
varieties of Arabic language. We perform a layer and neuron analysis on the
models using three intrinsic tasks: two morphological tagging tasks based on
MSA (modern standard Arabic) and dialectal POS-tagging and a dialectal
identification task. Our analysis enlightens interesting findings such as: i)
word morphology is learned at the lower and middle layers ii) dialectal
identification necessitate more knowledge and hence preserved even in the final
layers, iii) despite a large overlap in their vocabulary, the MSA-based models
fail to capture the nuances of Arabic dialects, iv) we found that neurons in
embedding layers are polysemous in nature, while the neurons in middle layers
are exclusive to specific properties.

本研究探究了基于不同阿拉伯语言方言预训练模型的内部表示，并使用三种内在任务对这些模型进行了层和神经元分析，包括两种基于 MSA 的形态标注任务和一种方言识别任务。研究发现，单词形态是在较低和中间层中学习的；方言识别需要更多的知识，因此即使在最终层中也需要保留；尽管词汇有很大的重叠，基于 MSA 的模型无法捕捉阿拉伯语方言的细微差别，而嵌入层中的神经元是多义的，而中间层中的神经元则专门用于特定的属性。