As a cornerstone in language modeling, tokenization involves segmenting text
inputs into pre-defined atomic units. Conventional statistical tokenizers often
disrupt constituent boundaries within words, thereby corrupting semantic
information. To address this drawback, we introduce morphological structure
guidance to tokenization and propose a deep model to induce character-level
structures of words. Specifically, the deep model jointly encodes internal
structures and representations of words with a mechanism named
$\textit{MorphOverriding}$ to ensure the indecomposability of morphemes. By
training the model with self-supervised objectives, our method is capable of
inducing character-level structures that align with morphological rules without
annotated training data. Based on the induced structures, our algorithm
tokenizes words through vocabulary matching in a top-down manner. Empirical
results indicate that the proposed method effectively retains complete
morphemes and outperforms widely adopted methods such as BPE and WordPiece on
both morphological segmentation tasks and language modeling tasks. The code
will be released later.

通过引入形态结构指导标记，提出了一种深度模型来诱导单词的字符级结构，该方法在形态分割任务和语言建模任务上表现良好，并优于 BPE 和 WordPiece 等广泛采用的方法。

无监督形态树分词器

Unsupervised Morphological Tree Tokenizer

Non-M\=aori-speaking New Zealanders (NMS)are able to segment M\=aori words in
a highlysimilar way to fluent speakers (Panther et al.,2024). This ability is
assumed to derive through the identification and extraction of statistically
recurrent forms. We examine this assumption by asking how NMS segmentations
compare to those produced by Morfessor, an unsupervised machine learning model
that operates based on statistical recurrence, across words formed by a variety
of morphological processes. Both NMS and Morfessor succeed in segmenting words
formed by concatenative processes (compounding and affixation without
allomorphy), but NMS also succeed for words that invoke templates
(reduplication and allomorphy) and other cues to morphological structure,
implying that their learning process is sensitive to more than just statistical
recurrence.

非毛利族新西兰人（NMS）在对毛利语的词汇分割方面与母语者表现高度相似，这种能力假设是通过识别和提取统计重复形式而获得的。我们通过比较 NMS 的分割结果与 Morfessor 的结果（一种基于统计重复的无监督机器学习模型）来验证这一假设，结果表明 NMS 在形态学过程中成功分割了通过合并过程（如复合词和无音位词缀）形成的词汇，同时他们的学习过程对于通过模板和其他形态学结构线索形成的词汇也敏感，说明 NMS 的学习过程不仅仅依赖于统计重复。

人类和机器对毛利语词素过程的无人监督学习：超越统计重复

More than Just Statistical Recurrence: Human and Machine Unsupervised  Learning of Māori Word Segmentation across Morphological Processes

Character-level models have become a popular approach specially for their
accessibility and ability to handle unseen data. However, little is known on
their ability to reveal the underlying morphological structure of a word, which
is a crucial skill for high-level semantic analysis tasks, such as semantic
role labeling (SRL). In this work, we train various types of SRL models that
use word, character and morphology level information and analyze how
performance of characters compare to words and morphology for several
languages. We conduct an in-depth error analysis for each morphological
typology and analyze the strengths and limitations of character-level models
that relate to out-of-domain data, training data size, long range dependencies
and model complexity. Our exhaustive analyses shed light on important
characteristics of character-level models and their semantic capability.

本文分析了使用字符、单词和形态水平信息的 SRL 模型和字符级模型的性能，研究表明字符级模型比单词和形态更适合处理未知数据并具有高层次的语义分析能力。