Simulation-free methods for training continuous-time generative models
construct probability paths that go between noise distributions and individual
data samples. Recent works, such as Flow Matching, derived paths that are
optimal for each data sample. However, these algorithms rely on independent
data and noise samples, and do not exploit underlying structure in the data
distribution for constructing probability paths. We propose Multisample Flow
Matching, a more general framework that uses non-trivial couplings between data
and noise samples while satisfying the correct marginal constraints. At very
small overhead costs, this generalization allows us to (i) reduce gradient
variance during training, (ii) obtain straighter flows for the learned vector
field, which allows us to generate high-quality samples using fewer function
evaluations, and (iii) obtain transport maps with lower cost in high
dimensions, which has applications beyond generative modeling. Importantly, we
do so in a completely simulation-free manner with a simple minimization
objective. We show that our proposed methods improve sample consistency on
downsampled ImageNet data sets, and lead to better low-cost sample generation.

通过非简单合并数据和噪声样本的 Multisample Flow Matching 算法，我们可以在完全不需要模拟的情况下，通过一个简单的最小化目标函数构造概率路径，实现连续时间生成模型训练参数、生成高质量样本和高维传输映射等目标的综合优化。

多样本流匹配：用小批量耦合直线化流

Multisample Flow Matching: Straightening Flows with Minibatch Couplings

Self-supervised learning, a.k.a., pretraining, is important in natural
language processing. Most of the pretraining methods first randomly mask some
positions in a sentence and then train a model to recover the tokens at the
masked positions. In such a way, the model can be trained without human
labeling, and the massive data can be used with billion parameters. Therefore,
the optimization efficiency becomes critical. In this paper, we tackle the
problem from the view of gradient variance reduction. In particular, we first
propose a principled gradient variance decomposition theorem, which shows that
the variance of the stochastic gradient of the language pretraining can be
naturally decomposed into two terms: the variance that arises from the sample
of data in a batch, and the variance that arises from the sampling of the mask.
The second term is the key difference between selfsupervised learning and
supervised learning, which makes the pretraining slower. In order to reduce the
variance of the second part, we leverage the importance sampling strategy,
which aims at sampling the masks according to a proposal distribution instead
of the uniform distribution. It can be shown that if the proposal distribution
is proportional to the gradient norm, the variance of the sampling is reduced.
To improve efficiency, we introduced a MAsk Proposal Network (MAPNet), which
approximates the optimal mask proposal distribution and is trained end-to-end
along with the model. According to the experimental result, our model converges
much faster and achieves higher performance than the baseline BERT model.

针对自监督学习在自然语言处理中的重要性，本文提出了一种基于梯度方差缩减的方法，通过引入重要性采样策略，利用 MAsk Proposal Network（MAPNet）进行优化，从而实现更高效的预训练并取得了比基础 BERT 模型更高的性能。