The ability to compare two degenerate probability distributions (i.e. two
probability distributions supported on two distinct low-dimensional manifolds
living in a much higher-dimensional space) is a crucial problem arising in the
estimation of generative models for high-dimensional observations such as those
arising in computer vision or natural language. It is known that optimal
transport metrics can represent a cure for this problem, since they were
specifically designed as an alternative to information divergences to handle
such problematic scenarios. Unfortunately, training generative machines using
OT raises formidable computational and statistical challenges, because of (i)
the computational burden of evaluating OT losses, (ii) the instability and lack
of smoothness of these losses, (iii) the difficulty to estimate robustly these
losses and their gradients in high dimension. This paper presents the first
tractable computational method to train large scale generative models using an
optimal transport loss, and tackles these three issues by relying on two key
ideas: (a) entropic smoothing, which turns the original OT loss into one that
can be computed using Sinkhorn fixed point iterations; (b) algorithmic
(automatic) differentiation of these iterations. These two approximations
result in a robust and differentiable approximation of the OT loss with
streamlined GPU execution. Entropic smoothing generates a family of losses
interpolating between Wasserstein (OT) and Maximum Mean Discrepancy (MMD), thus
allowing to find a sweet spot leveraging the geometry of OT and the favorable
high-dimensional sample complexity of MMD which comes with unbiased gradient
estimates. The resulting computational architecture complements nicely standard
deep network generative models by a stack of extra layers implementing the loss
function.

本文介绍了一种使用最优输送损失的可行计算方法，通过熵平滑和自动微分来减少计算负担、提高稳定性和平滑性，获得鲁棒和可微分的最优输送损失的逼近，从而训练大规模生成模型并补充标准深度网络生成模型的计算机架构。

使用 Sinkhorn 散度学习生成模型

Learning Generative Models with Sinkhorn Divergences

We propose a novel method for semi-supervised learning (SSL) based on
data-driven distributionally robust optimization (DRO) using optimal transport
metrics. Our proposed method enhances generalization error by using the
unlabeled data to restrict the support of the worst case distribution in our
DRO formulation. We enable the implementation of our DRO formulation by
proposing a stochastic gradient descent algorithm which allows to easily
implement the training procedure. We demonstrate that our Semi-supervised DRO
method is able to improve the generalization error over natural supervised
procedures and state-of-the-art SSL estimators. Finally, we include a
discussion on the large sample behavior of the optimal uncertainty region in
the DRO formulation. Our discussion exposes important aspects such as the role
of dimension reduction in SSL.

提出了一种基于数据驱动的分布式鲁棒优化和最优运输度量的半监督学习方法，通过使用未标记的数据来限制最坏情况分布的支持，从而提高一般化误差，并证明该方法能够改善一般化误差，具有重要的大样本行为，特别是在半监督学习中降维的作用。

基于分布鲁棒优化的半监督学习

Semi-supervised Learning based on Distributionally Robust Optimization

We study the problem of estimating, in the sense of optimal transport
metrics, a measure which is assumed supported on a manifold embedded in a
Hilbert space. By establishing a precise connection between optimal transport
metrics, optimal quantization, and learning theory, we derive new probabilistic
bounds for the performance of a classic algorithm in unsupervised learning
(k-means), when used to produce a probability measure derived from the data. In
the course of the analysis, we arrive at new lower bounds, as well as
probabilistic upper bounds on the convergence rate of the empirical law of
large numbers, which, unlike existing bounds, are applicable to a wide class of
measures.

通过优化传输度量，在嵌入 Hilbert 空间的流形上估计一种衡量方法，并将量化优化和学习理论联系起来，为无监督学习中经典算法（k-means）的性能提供新的概率界限。在分析的过程中，我们得出了新的下界和概率上界，这些上下界适用于广泛的测度范围。