Common designs of model evaluation typically focus on monolingual settings,
where different models are compared according to their performance on a single
data set that is assumed to be representative of all possible data for the task
at hand. While this may be reasonable for a large data set, this assumption is
difficult to maintain in low-resource scenarios, where artifacts of the data
collection can yield data sets that are outliers, potentially making
conclusions about model performance coincidental. To address these concerns, we
investigate model generalizability in crosslinguistic low-resource scenarios.
Using morphological segmentation as the test case, we compare three broad
classes of models with different parameterizations, taking data from 11
languages across 6 language families. In each experimental setting, we evaluate
all models on a first data set, then examine their performance consistency when
introducing new randomly sampled data sets with the same size and when applying
the trained models to unseen test sets of varying sizes. The results
demonstrate that the extent of model generalization depends on the
characteristics of the data set, and does not necessarily rely heavily on the
data set size. Among the characteristics that we studied, the ratio of morpheme
overlap and that of the average number of morphemes per word between the
training and test sets are the two most prominent factors. Our findings suggest
that future work should adopt random sampling to construct data sets with
different sizes in order to make more responsible claims about model
evaluation.

该研究旨在通过使用多种语言的数据，比较不同参数设置下的模型表现，从而证明模型通用性取决于数据集的特征而不只是数据集的大小。