The Information Bottleneck (IB) principle offers an information-theoretic
framework for analyzing the training process of deep neural networks (DNNs).
Its essence lies in tracking the dynamics of two mutual information (MI)
values: one between the hidden layer and the class label, and the other between
the hidden layer and the DNN input. According to the hypothesis put forth by
Shwartz-Ziv and Tishby (2017), the training process consists of two distinct
phases: fitting and compression. The latter phase is believed to account for
the good generalization performance exhibited by DNNs. Due to the challenging
nature of estimating MI between high-dimensional random vectors, this
hypothesis has only been verified for toy NNs or specific types of NNs, such as
quantized NNs and dropout NNs. In this paper, we introduce a comprehensive
framework for conducting IB analysis of general NNs. Our approach leverages the
stochastic NN method proposed by Goldfeld et al. (2019) and incorporates a
compression step to overcome the obstacles associated with high dimensionality.
In other words, we estimate the MI between the compressed representations of
high-dimensional random vectors. The proposed method is supported by both
theoretical and practical justifications. Notably, we demonstrate the accuracy
of our estimator through synthetic experiments featuring predefined MI values.
Finally, we perform IB analysis on a close-to-real-scale convolutional DNN,
which reveals new features of the MI dynamics.

该研究利用 Information Bottleneck 原理和深度神经网络，采用互信息和压缩等技术，针对高维随机向量实现了对一般深度神经网络的信息瓶颈分析，并在一个近实际规模的卷积深度神经网络上揭示了互信息动态的新特征。

深度神经网络的信息瓶颈分析通过有损压缩

Information Bottleneck Analysis of Deep Neural Networks via Lossy  Compression

The Sliced-Wasserstein distance (SW) is being increasingly used in machine
learning applications as an alternative to the Wasserstein distance and offers
significant computational and statistical benefits. Since it is defined as an
expectation over random projections, SW is commonly approximated by Monte
Carlo. We adopt a new perspective to approximate SW by making use of the
concentration of measure phenomenon: under mild assumptions, one-dimensional
projections of a high-dimensional random vector are approximately Gaussian.
Based on this observation, we develop a simple deterministic approximation for
SW. Our method does not require sampling a number of random projections, and is
therefore both accurate and easy to use compared to the usual Monte Carlo
approximation. We derive nonasymptotical guarantees for our approach, and show
that the approximation error goes to zero as the dimension increases, under a
weak dependence condition on the data distribution. We validate our theoretical
findings on synthetic datasets, and illustrate the proposed approximation on a
generative modeling problem.

通过利用测度集中现象，我们基于高维随机向量的一维投影近似 Sliced-Wasserstein 距离，避免了通常需要的蒙特卡罗模拟，这一方法简单精确，用于生成模型问题效果显著。

利用随机投影之浓度快速近似计算切片 - 瓦热斯坦距离

Fast Approximation of the Sliced-Wasserstein Distance Using  Concentration of Random Projections

This paper studies the asymptotic behaviors of the pairwise angles among n
randomly and uniformly distributed unit vectors in R^p as the number of points
n -> infinity, while the dimension p is either fixed or growing with n. For
both settings, we derive the limiting empirical distribution of the random
angles and the limiting distributions of the extreme angles. The results reveal
interesting differences in the two settings and provide a precise
characterization of the folklore that "all high-dimensional random vectors are
almost always nearly orthogonal to each other". Applications to statistics and
machine learning and connections with some open problems in physics and
mathematics are also discussed.

研究在高维空间中，随着数据元素数量不断增加，单位向量之间角度的极限分布，该研究对高维空间向量的统计学和机器学习应用有指导意义。