We propose an efficient modeling framework for cross-lingual named entity
recognition in semi-structured text data. Our approach relies on both knowledge
distillation and consistency training. The modeling framework leverages
knowledge from a large language model (XLMRoBERTa) pre-trained on the source
language, with a student-teacher relationship (knowledge distillation). The
student model incorporates unsupervised consistency training (with KL
divergence loss) on the low-resource target language.
We employ two independent datasets of SMSs in English and Arabic, each
carrying semi-structured banking transaction information, and focus on
exhibiting the transfer of knowledge from English to Arabic. With access to
only 30 labeled samples, our model can generalize the recognition of merchants,
amounts, and other fields from English to Arabic. We show that our modeling
approach, while efficient, performs best overall when compared to
state-of-the-art approaches like DistilBERT pre-trained on the target language
or a supervised model directly trained on labeled data in the target language.
Our experiments show that it is enough to learn to recognize entities in
English to reach reasonable performance in a low-resource language in the
presence of a few labeled samples of semi-structured data. The proposed
framework has implications for developing multi-lingual applications,
especially in geographies where digital endeavors rely on both English and one
or more low-resource language(s), sometimes mixed with English or employed
singly.

我们提出了一个高效的建模框架，用于半结构化文本数据中的跨语言命名实体识别。我们的方法依赖于知识蒸馏和一致性训练，并且利用预训练在源语言上的大型语言模型（XLMRoBERTa）的知识，通过学生 - 教师关系（知识蒸馏）。学生模型在低资源目标语言上采用无监督的一致性训练（使用 KL 散度损失）。我们使用两个独立的 SMS 数据集，分别是英语和阿拉伯语，每个数据集都携带有半结构化的银行交易信息，并侧重展示从英语到阿拉伯语的知识传递。通过仅使用 30 个标记样本，我们的模型可以将商家、金额和其他字段的识别从英语推广到阿拉伯语。我们的实验表明，在低资源语言的少量标记样本的情况下，学习在英语中识别实体就足以在低资源语言中达到合理的性能。该提出的框架对于开发多语言应用程序，尤其是在数字化努力依赖英语和一种或多种低资源语言的地理区域具有重要意义，有时与英语混合使用或独立使用。