Most pretrained language models rely on subword tokenization, which processes
text as a sequence of subword tokens. However, different granularities of text,
such as characters, subwords, and words, can contain different kinds of
information. Previous studies have shown that incorporating multiple input
granularities improves model generalization, yet very few of them outputs
useful representations for each granularity. In this paper, we introduce the
entanglement model, aiming to combine character and subword language models.
Inspired by vision-language models, our model treats characters and subwords as
separate modalities, and it generates mutually informed representations for
both granularities as output. We evaluate our model on text classification,
named entity recognition, and POS-tagging tasks. Notably, the entanglement
model outperforms its backbone language models, particularly in the presence of
noisy texts and low-resource languages. Furthermore, the entanglement model
even outperforms larger pre-trained models on all English sequence labeling
tasks and classification tasks. Our anonymized code is available at
this https URL

该研究使用预训练语言模型和交织模型，在处理不同粒度的文本时，提高了模型的泛化能力并在多个 NLP 任务中取得了优秀的性能。