We introduce an innovative and mathematically rigorous definition for
computing common information from multi-view data, drawing inspiration from
G\'acs-K\"orner common information in information theory. Leveraging this
definition, we develop a novel supervised multi-view learning framework to
capture both common and unique information. By explicitly minimizing a total
correlation term, the extracted common information and the unique information
from each view are forced to be independent of each other, which, in turn,
theoretically guarantees the effectiveness of our framework. To estimate
information-theoretic quantities, our framework employs matrix-based
R{\'e}nyi's $\alpha$-order entropy functional, which forgoes the need for
variational approximation and distributional estimation in high-dimensional
space. Theoretical proof is provided that our framework can faithfully discover
both common and unique information from multi-view data. Experiments on
synthetic and seven benchmark real-world datasets demonstrate the superior
performance of our proposed framework over state-of-the-art approaches.

提出了一个创新且数学严谨的定义，用于从多视角数据中计算共有信息。利用这个定义，开发了一种新颖的监督式多视角学习框架，以捕捉共有信息和独特信息。通过明确最小化总相关性项，可以让从每个视角提取的共有信息和独特信息彼此独立，从而在理论上保证了我们框架的有效性。通过基于矩阵的 Rényi's $α$ 阶熵函数来估计信息论量，避免了在高维空间中进行变分逼近和分布估计的需求。理论证明了我们的框架可以忠实地发现多视角数据中的共有信息和独特信息。在合成数据和七个标准数据集上的实验证明了我们提出的框架优于现有方法的性能。

多视图数据中的共享信息发现

Discovering Common Information in Multi-view Data

In most applications of utilizing neural networks for mathematical
optimization, a dedicated model is trained for each specific optimization
objective. However, in many scenarios, several distinct yet correlated
objectives or tasks often need to be optimized on the same set of problem
inputs. Instead of independently training a different neural network for each
problem separately, it would be more efficient to exploit the correlations
between these objectives and to train multiple neural network models with
shared model parameters and feature representations. To achieve this, this
paper first establishes the concept of common information: the shared knowledge
required for solving the correlated tasks, then proposes a novel approach for
model training by adding into the model an additional reconstruction stage
associated with a new reconstruction loss. This loss is for reconstructing the
common information starting from a selected hidden layer in the model. The
proposed approach encourages the learned features to be general and
transferable, and therefore can be readily used for efficient transfer
learning. For numerical simulations, three applications are studied: transfer
learning on classifying MNIST handwritten digits, the device-to-device wireless
network power allocation, and the multiple-input-single-output network downlink
beamforming and localization. Simulation results suggest that the proposed
approach is highly efficient in data and model complexity, is resilient to
over-fitting, and has competitive performances.

该论文提出了一种利用神经网络进行优化的方法，该方法通过共享模型参数和特征表示来训练多个神经网络模型，以解决多个相关任务。论文引入了共同信息的概念，并提出了一种新的模型训练方法，通过添加一个额外的重构阶段和重构损失，将共同信息重新构建出来。该方法鼓励学习到的特征具有通用性和可迁移性，因此可以用于有效的迁移学习。通过数值模拟，研究了在不同应用领域上的性能表现。结果表明，该方法在数据和模型复杂度方面非常高效，能够抵抗过拟合，并且具有竞争性能。

带有重建损失的迁移学习

Transfer Learning with Reconstruction Loss

Measuring the relationship between any pair of variables is a rich and active
area of research that is central to scientific practice. In contrast,
characterizing the common information among any group of variables is typically
a theoretical exercise with few practical methods for high-dimensional data. A
promising solution would be a multivariate generalization of the famous Wyner
common information, but this approach relies on solving an apparently
intractable optimization problem. We leverage the recently introduced
information sieve decomposition to formulate an incremental version of the
common information problem that admits a simple fixed point solution, fast
convergence, and complexity that is linear in the number of variables. This
scalable approach allows us to demonstrate the usefulness of common information
in high-dimensional learning problems. The sieve outperforms standard methods
on dimensionality reduction tasks, solves a blind source separation problem
that cannot be solved with ICA, and accurately recovers structure in brain
imaging data.

本文提出了一种可扩展的方法，通过信息筛分解和简单的迭代过程，解决 Wyner 公共信息问题。该方法在高维度学习问题上比标准方法表现更好，并成功地恢复了大脑成像数据中的结构。