Choosing an appropriate tokenization scheme is often a bottleneck in
low-resource cross-lingual transfer. To understand the downstream implications
of text representation choices, we perform a comparative analysis on language
models having diverse text representation modalities including 2
segmentation-based models (\texttt{BERT}, \texttt{mBERT}), 1 image-based model
(\texttt{PIXEL}), and 1 character-level model (\texttt{CANINE}). First, we
propose a scoring Language Quotient (LQ) metric capable of providing a weighted
representation of both zero-shot and few-shot evaluation combined. Utilizing
this metric, we perform experiments comprising 19 source languages and 133
target languages on three tasks (POS tagging, Dependency parsing, and NER). Our
analysis reveals that image-based models excel in cross-lingual transfer when
languages are closely related and share visually similar scripts. However, for
tasks biased toward word meaning (POS, NER), segmentation-based models prove to
be superior. Furthermore, in dependency parsing tasks where word relationships
play a crucial role, models with their character-level focus, outperform
others. Finally, we propose a recommendation scheme based on our findings to
guide model selection according to task and language requirements.

选择一个合适的分词方案通常是低资源跨语言转移的瓶颈。为了理解文本表示选择的后续影响，我们对具有不同文本表示模式的语言模型进行了比较分析，包括 2 个基于分词的模型（BERT，mBERT）、一个基于图像的模型（PIXEL）和一个基于字符级别的模型（CANINE）。通过提出的评分语言系数（LQ），我们在 19 种源语言和 133 种目标语言上执行了涵盖 POS 标记、依赖解析和 NER 的实验证明了基于图像的模型在语言密切相关且具有相似视觉脚本的情况下，在跨语言转移中表现出色。然而，在偏向于词义（POS，NER）的任务中，基于分词的模型表现优越。此外，在强依赖于词汇关系的依赖解析任务中，具有字符级别关注的模型胜过其他模型。最后，根据我们的发现，我们提出了一个基于我们的研究成果的建议方案，以指导根据任务和语言要求进行模型选择。

是否对文本进行标记化：跨语言转移的文本表示的比较研究

To token or not to token: A Comparative Study of Text Representations  for Cross-Lingual Transfer

Despite recent successes in language models, their ability to represent
numbers is insufficient. Humans conceptualize numbers based on their
magnitudes, effectively projecting them on a number line; whereas subword
tokenization fails to explicitly capture magnitude by splitting numbers into
arbitrary chunks. To alleviate this shortcoming, alternative approaches have
been proposed that modify numbers at various stages of the language modeling
pipeline. These methods change either the (1) notation in which numbers are
written (\eg scientific vs decimal), the (2) vocabulary used to represent
numbers or the entire (3) architecture of the underlying language model, to
directly regress to a desired number.
Previous work suggests that architectural change helps achieve
state-of-the-art on number estimation but we find an insightful ablation:
changing the model's vocabulary instead (\eg introduce a new token for numbers
in range 10-100) is a far better trade-off. In the context of masked number
prediction, a carefully designed tokenization scheme is both the simplest to
implement and sufficient, \ie with similar performance to the state-of-the-art
approach that requires making significant architectural changes. Finally, we
report similar trends on the downstream task of numerical fact estimation (for
Fermi Problems) and discuss reasons behind our findings.

改变语言模型的词汇表，而不是架构，可以更好地处理数字；在预测掩码数字和数值事实估计方面具有相似的性能，这要归功于仔细设计的标记化方案。