Standard autoregressive seq2seq models are easily trained by max-likelihood,
but tend to show poor results under small-data conditions. We introduce a class
of seq2seq models, GAMs (Global Autoregressive Models), which combine an
autoregressive component with a log-linear component, allowing the use of
global \textit{a priori} features to compensate for lack of data. We train
these models in two steps. In the first step, we obtain an \emph{unnormalized}
GAM that maximizes the likelihood of the data, but is improper for fast
inference or evaluation. In the second step, we use this GAM to train (by
distillation) a second autoregressive model that approximates the
\emph{normalized} distribution associated with the GAM, and can be used for
fast inference and evaluation. Our experiments focus on language modelling
under synthetic conditions and show a strong perplexity reduction of using the
second autoregressive model over the standard one.

介绍了一个基于 seq2seq 模型的 GAM（Global Autoregressive Models）的类，将自回归组件与对数线性组件相结合，在小数据条件下使用全局先验特征来补偿数据不足，并使用两个步骤来训练此模型，实验表明，使用第二个自回归模型可以强有力地减少混乱度。