Tokenizing raw texts into word units is an essential pre-processing step for
critical tasks in the NLP pipeline such as tagging, parsing, named entity
recognition, and more. For most languages, this tokenization step
straightforward. However, for languages with high token-internal complexity,
further token-to-word segmentation is required. Previous canonical segmentation
studies were based on character-level frameworks, with no contextualised
representation involved. Contextualized vectors a la BERT show remarkable
results in many applications, but were not shown to improve performance on
linguistic segmentation per se. Here we propose a novel neural segmentation
model which combines the best of both worlds, contextualised token
representation and char-level decoding, which is particularly effective for
languages with high token-internal complexity and extreme morphological
ambiguity. Our model shows substantial improvements in segmentation accuracy on
Hebrew and Arabic compared to the state-of-the-art, and leads to further
improvements on downstream tasks such as Part-of-Speech Tagging, Dependency
Parsing and Named-Entity Recognition, over existing pipelines. When comparing
our segmentation-first pipeline with joint segmentation and labeling in the
same settings, we show that, contrary to pre-neural studies, the pipeline
performance is superior.

提出了一种结合了上下文化词元表示和字符级解码的神经分割模型，对于具有高令牌内部复杂性和极度形态歧义的希伯来语和阿拉伯语等语言，其分割准确性得到了显着提高，并对下游 NLP 任务的表现进行优化。