We explore the possibility of meta-learning for the language-independent
unsupervised tokenization problem for English, Russian, and Chinese. We
implement the meta-learning approach for automatic determination of
hyper-parameters of the unsupervised tokenization model proposed in earlier
works, relying on various human-independent fitness functions such as
normalised anti-entropy, compression factor and cross-split F 1 score, as well
as additive and multiplicative composite combinations of the three metrics,
testing them against the conventional F1 tokenization score. We find a fairly
good correlation between the latter and the additive combination of the former
three metrics for English and Russian. In case of Chinese, we find a
significant correlation between the F 1 score and the compression factor. Our
results suggest the possibility of robust unsupervised tokenization of
low-resource and dead languages and allow us to think about human languages in
terms of the evolution of efficient symbolic communication codes with different
structural optimisation schemes that have evolved in different human cultures.

该研究探讨了对于英语、俄语和汉语的无监督分词问题可行的元学习方法，通过实现基于不同人类独立健身函数的元学习方法以及三个度量的加性和乘性组合来测试不同语言的无监督分词模型并找到了适合每种语言的最佳设置。

无监督跨语言分词的自调参数

Self-tuning hyper-parameters for unsupervised cross-lingual tokenization

In the presented study, we discover that the so-called "transition freedom"
metric appears superior for unsupervised tokenization purposes in comparison to
statistical metrics such as mutual information and conditional probability,
providing F-measure scores in range from 0.71 to 1.0 across explored
multilingual corpora. We find that different languages require different
offshoots of that metric (such as derivative, variance, and "peak values") for
successful tokenization. Larger training corpora do not necessarily result in
better tokenization quality, while compressing the models by eliminating
statistically weak evidence tends to improve performance. The proposed
unsupervised tokenization technique provides quality better than or comparable
to lexicon-based ones, depending on the language.

本研究发现，相比于互信息和条件概率等统计度量方法，所谓的 “过渡自由度” 度量方法在无监督分词方面表现更佳，跨多语料库提供了 0.71 到 1.0 的 F - 度量分数。我们发现，不同的语言需要该度量方法的不同分支（如导数、方差和 “峰值”）才能实现成功的分词。 更大的训练语料库不一定会导致更好的分词质量，而通过消除统计上薄弱的证据来压缩模型往往会提高性能。 该无监督分词技术提供的质量优于或与基于词典的技术相媲美，具体效果取决于语言。