Self-training is one of the earliest and simplest semi-supervised methods.
The key idea is to augment the original labeled dataset with unlabeled data
paired with the model's prediction (i.e. the pseudo-parallel data). While
self-training has been extensively studied on classification problems, in
complex sequence generation tasks (e.g. machine translation) it is still
unclear how self-training works due to the compositionality of the target
space. In this work, we first empirically show that self-training is able to
decently improve the supervised baseline on neural sequence generation tasks.
Through careful examination of the performance gains, we find that the
perturbation on the hidden states (i.e. dropout) is critical for self-training
to benefit from the pseudo-parallel data, which acts as a regularizer and
forces the model to yield close predictions for similar unlabeled inputs. Such
effect helps the model correct some incorrect predictions on unlabeled data. To
further encourage this mechanism, we propose to inject noise to the input
space, resulting in a "noisy" version of self-training. Empirical study on
standard machine translation and text summarization benchmarks shows that noisy
self-training is able to effectively utilize unlabeled data and improve the
performance of the supervised baseline by a large margin.

本文提出了一种基于自训练的半监督学习方法，通过加入 label 缺失的无标签数据和模型预测，模型的性能和精确度可以得到有效的提升，而 dropout 和注入噪声则是关键操作。

神经序列生成的自训练再探

Revisiting Self-Training for Neural Sequence Generation

Neural sequence generation is typically performed token-by-token and
left-to-right. Whenever a token is generated only previously produced tokens
are taken into consideration. In contrast, for problems such as sequence
classification, bidirectional attention, which takes both past and future
tokens into consideration, has been shown to perform much better. We propose to
make the sequence generation process bidirectional by employing special
placeholder tokens. Treated as a node in a fully connected graph, a placeholder
token can take past and future tokens into consideration when generating the
actual output token. We verify the effectiveness of our approach experimentally
on two conversational tasks where the proposed bidirectional model outperforms
competitive baselines by a large margin.

这篇论文提出了一种采用特殊占位符令序列生成过程具有双向性的方法，将生成过程视为一个完全连接图，有效提高了对话任务的性能。

双向序列生成中关注未来标记

Attending to Future Tokens For Bidirectional Sequence Generation

Modern neural sequence generation models are built to either generate tokens
step-by-step from scratch or (iteratively) modify a sequence of tokens bounded
by a fixed length. In this work, we develop Levenshtein Transformer, a new
partially autoregressive model devised for more flexible and amenable sequence
generation. Unlike previous approaches, the atomic operations of our model are
insertion and deletion. The combination of them facilitates not only generation
but also sequence refinement allowing dynamic length changes. We also propose a
set of new training techniques dedicated at them, effectively exploiting one as
the other's learning signal thanks to their complementary nature. Experiments
applying the proposed model achieve comparable performance but much-improved
efficiency on both generation (e.g. machine translation, text summarization)
and refinement tasks (e.g. automatic post-editing). We further confirm the
flexibility of our model by showing a Levenshtein Transformer trained by
machine translation can straightforwardly be used for automatic post-editing.

本文提出了一种部分自回归模型 Levenshtein Transformer，在线性时间复杂度内支持插入和删除操作，实现了互补的拆解和精细化的设计思路，成功地提高了机器翻译和文本自动编辑的效率。