Self-supervised neural machine translation (SSNMT) jointly learns to identify
and select suitable training data from comparable (rather than parallel)
corpora and to translate, in a way that the two tasks support each other in a
virtuous circle. In this study, we provide an in-depth analysis of the sampling
choices the SSNMT model makes during training. We show how, without it having
been told to do so, the model self-selects samples of increasing (i) complexity
and (ii) task-relevance in combination with (iii) performing a denoising
curriculum. We observe that the dynamics of the mutual-supervision signals of
both system internal representation types are vital for the extraction and
translation performance. We show that in terms of the Gunning-Fog Readability
index, SSNMT starts extracting and learning from Wikipedia data suitable for
high school students and quickly moves towards content suitable for first year
undergraduate students.

该研究提供了有关 SSNMT 模型在训练期间进行采样选择的深入分析，证明了该模型能够自我选择越来越复杂和相关任务的样本，并执行去噪曲线，以提高提取和翻译性能。其中，相互监督信号的动态对提取和翻译性能至关重要。研究显示，使用 Gunning-Fog 可读性指数时，SSNMT 从适合高中学生的 Wikipedia 数据开始提取和学习，并迅速转向适合大一学生的内容。