Data-driven subword segmentation has become the default strategy for open-vocabulary machine translation and other NLP tasks, but may not be sufficiently generic for optimal learning of non-concatenative morphology. We design a test suite to evaluate segmentation strategies on different types of morphological phenomena in a controlled, semi-synthetic setting. In our experiments, we compare how well machine translation models trained on subword- and character-level can translate these morphological phenomena. We find that learning to analyse and generate morphologically complex surface representations is still challenging, especially for non-concatenative morphological phenomena like reduplication or vowel harmony and for rare word stems. Based on our results, we recommend that novel text representation strategies be tested on a range of typologically diverse languages to minimise the risk of adopting a strategy that inadvertently disadvantages certain languages.

本论文研究基于数据驱动的子词分割在自然语言处理等领域的应用是否适用于非连接语素构词方式，通过一套测试套件对分割策略在不同形态学现象上的表现进行比较，并发现学习分析和生成表面的形态上下文仍然具有挑战性且应针对多种语言应用测试新的文本表征策略，以减少不同策略对某些语言造成的不利影响。

子词切分对非拼接形态转换的适用性如何？