In this work we present a new method for the estimation of Mutual Information
(MI) between random variables. Our approach is based on an original
interpretation of the Girsanov theorem, which allows us to use score-based
diffusion models to estimate the Kullback Leibler divergence between two
densities as a difference between their score functions. As a by-product, our
method also enables the estimation of the entropy of random variables. Armed
with such building blocks, we present a general recipe to measure MI, which
unfolds in two directions: one uses conditional diffusion process, whereas the
other uses joint diffusion processes that allow simultaneous modelling of two
random variables. Our results, which derive from a thorough experimental
protocol over all the variants of our approach, indicate that our method is
more accurate than the main alternatives from the literature, especially for
challenging distributions. Furthermore, our methods pass MI self-consistency
tests, including data processing and additivity under independence, which
instead are a pain-point of existing methods.

通过基于 Girsanov 定理的新方法，我们提出了一种估计随机变量之间互信息（MI）的方法。我们的方法基于分数函数的扩散模型来估计两个密度之间的 Kullback Leibler 散度，并衍生出估计随机变量熵的方法。我们的结果表明，我们的方法在挑战性分布情况下比文献中的主要替代方法更准确，并通过自洽性测试，包括数据处理和独立性下的可加性。

MINDE: 互信息神经扩散估计

MINDE: Mutual Information Neural Diffusion Estimation

Variational Autoencoders (VAEs) have become a cornerstone in generative
modeling and representation learning within machine learning. This paper
explores a nuanced aspect of VAEs, focusing on interpreting the Kullback
Leibler (KL) Divergence, a critical component within the Evidence Lower Bound
(ELBO) that governs the trade-off between reconstruction accuracy and
regularization. While the KL Divergence enforces alignment between latent
variable distributions and a prior imposing a structure on the overall latent
space but leaves individual variable distributions unconstrained. The proposed
method redefines the ELBO with a mixture of Gaussians for the posterior
probability, introduces a regularization term to prevent variance collapse, and
employs a PatchGAN discriminator to enhance texture realism. Implementation
details involve ResNetV2 architectures for both the Encoder and Decoder. The
experiments demonstrate the ability to generate realistic faces, offering a
promising solution for enhancing VAE based generative models.

该论文探讨了变分自编码器（VAEs）的一个微妙方面，着重解释了 Kullback Leibler（KL）散度，这是 Evidence Lower Bound（ELBO）中的一个关键组成部分，用于平衡重构准确性和正则化之间的权衡。通过使用混合高斯后验概率重新定义 ELBO，引入正则化项以防止方差崩溃，并使用 PatchGAN 鉴别器增强纹理真实性，该方法在实现细节中采用了 ResNetV2 架构用于编码器和解码器。实验证明其能够生成逼真的人脸，为增强基于 VAE 的生成模型提供了有希望的解决方案。

GAMIX-VAE: 基于高斯混合后验的 VAE

GAMIX-VAE: A VAE with Gaussian Mixture Based Posterior

Distributionally Robust Optimization (DRO), as a popular method to train
robust models against distribution shift between training and test sets, has
received tremendous attention in recent years. In this paper, we propose and
analyze stochastic algorithms that apply to both non-convex and convex losses
for solving Kullback Leibler divergence constrained DRO problem. Compared with
existing methods solving this problem, our stochastic algorithms not only enjoy
competitive if not better complexity independent of sample size but also just
require a constant batch size at every iteration, which is more practical for
broad applications. We establish a nearly optimal complexity bound for finding
an $\epsilon$ stationary solution for non-convex losses and an optimal
complexity for finding an $\epsilon$ optimal solution for convex losses.
Empirical studies demonstrate the effectiveness of the proposed algorithms for
solving non-convex and convex constrained DRO problems.

本文提出并分析了一种基于随机算法的方法，用于解决 Kullback Leibler divergence 约束的 Distributionally Robust Optimization 问题，该方法适用于非凸和凸损失函数，并具有更高的竞争性和更实用的常数批量大小迭代复杂度。