Mixture models trained via EM are among the simplest, most widely used and
well understood latent variable models in the machine learning literature.
Surprisingly, these models have been hardly explored in text generation
applications such as machine translation. In principle, they provide a latent
variable to control generation and produce a diverse set of hypotheses. In
practice, however, mixture models are prone to degeneracies---often only one
component gets trained or the latent variable is simply ignored. We find that
disabling dropout noise in responsibility computation is critical to successful
training. In addition, the design choices of parameterization, prior
distribution, hard versus soft EM and online versus offline assignment can
dramatically affect model performance. We develop an evaluation protocol to
assess both quality and diversity of generations against multiple references,
and provide an extensive empirical study of several mixture model variants. Our
analysis shows that certain types of mixture models are more robust and offer
the best trade-off between translation quality and diversity compared to
variational models and diverse decoding approaches.\footnote{Code to reproduce
the results in this paper is available at
https://github.com/pytorch/fairseq}

本文研究了通过 EM 训练的混合模型在文本生成方面的应用，通过设计参数、优化模型选择和评估协议，提出了一些更为鲁棒的混合模型，相比于变分模型和多样的解码方法在翻译质量和多样性之间提供了更好的平衡点。

多样化机器翻译的混合模型：实用技巧

Mixture Models for Diverse Machine Translation: Tricks of the Trade

We consider approximate maximum likelihood parameter estimation in nonlinear
state-space models. We discuss both direct optimization of the likelihood and
expectation--maximization (EM). For EM, we also give closed-form expressions
for the maximization step in a class of models that are linear in parameters
and have additive noise. To obtain approximations to the filtering and
smoothing distributions needed in the likelihood-maximization methods, we focus
on using Gaussian filtering and smoothing algorithms that employ sigma-points
to approximate the required integrals. We discuss different sigma-point schemes
based on the third, fifth, seventh, and ninth order unscented transforms and
the Gauss--Hermite quadrature rule. We compare the performance of the methods
in two simulated experiments: a univariate nonlinear growth model as well as
tracking of a maneuvering target. In the experiments, we also compare against
approximate likelihood estimates obtained by particle filtering and extended
Kalman filtering based methods. The experiments suggest that the higher-order
unscented transforms may in some cases provide more accurate estimates

研究用于非线性状态空间模型的近似极大似然参数估计，讨论了直接最大化似然和 EM 算法，使用 sigma 点来近似所需积分的高斯滤波和平滑算法，比较了不同阶的 unscented 变换以及高斯 - 埃尔米特积分规则得出的方法在两个模拟实验中的表现

基于卡尔曼滤波器和平滑器的非线性动态系统参数估计

Sigma-Point Filtering and Smoothing Based Parameter Estimation in  Nonlinear Dynamic Systems

Marginal MAP problems are notoriously difficult tasks for graphical models.
We derive a general variational framework for solving marginal MAP problems, in
which we apply analogues of the Bethe, tree-reweighted, and mean field
approximations. We then derive a "mixed" message passing algorithm and a
convergent alternative using CCCP to solve the BP-type approximations.
Theoretically, we give conditions under which the decoded solution is a global
or local optimum, and obtain novel upper bounds on solutions. Experimentally we
demonstrate that our algorithms outperform related approaches. We also show
that EM and variational EM comprise a special case of our framework.

本文针对图像模型的边缘 MAP 问题，提出了一种变分框架，运用 Bethe、tree-reweighted、mean field 逼近等方法得出 mixed message passing algorithm 和使用 CCCP 解决 BP-type 逼近的算法，并在实验中证明了我们的算法优于相关方法。此外，本文还表明 EM 和 variational EM 组成我们框架的一个特殊情况。