We present new results to model and understand the role of encoder-decoder
design in machine learning (ML) from an information-theoretic angle. We use two
main information concepts, information sufficiency (IS) and mutual information
loss (MIL), to represent predictive structures in machine learning. Our first
main result provides a functional expression that characterizes the class of
probabilistic models consistent with an IS encoder-decoder latent predictive
structure. This result formally justifies the encoder-decoder forward stages
many modern ML architectures adopt to learn latent (compressed) representations
for classification. To illustrate IS as a realistic and relevant model
assumption, we revisit some known ML concepts and present some interesting new
examples: invariant, robust, sparse, and digital models. Furthermore, our IS
characterization allows us to tackle the fundamental question of how much
performance (predictive expressiveness) could be lost, using the cross entropy
risk, when a given encoder-decoder architecture is adopted in a learning
setting. Here, our second main result shows that a mutual information loss
quantifies the lack of expressiveness attributed to the choice of a (biased)
encoder-decoder ML design. Finally, we address the problem of universal
cross-entropy learning with an encoder-decoder design where necessary and
sufficiency conditions are established to meet this requirement. In all these
results, Shannon's information measures offer new interpretations and
explanations for representation learning.

我们从信息论的角度提出了有关编码器 - 解码器设计在机器学习中发挥作用的新结果，使用信息充分性和互信息损失来表示机器学习中的预测结构，并使用香农的信息度量为表示学习提供了新的解释。

机器学习中编码解码结构的信息度量理解

Understanding Encoder-Decoder Structures in Machine Learning Using  Information Measures

In many real-world applications, face recognition models often degenerate
when training data (referred to as source domain) are different from testing
data (referred to as target domain). To alleviate this mismatch caused by some
factors like pose and skin tone, the utilization of pseudo-labels generated by
clustering algorithms is an effective way in unsupervised domain adaptation.
However, they always miss some hard positive samples. Supervision on
pseudo-labeled samples attracts them towards their prototypes and would cause
an intra-domain gap between pseudo-labeled samples and the remaining unlabeled
samples within target domain, which results in the lack of discrimination in
face recognition. In this paper, considering the particularity of face
recognition, we propose a novel adversarial information network (AIN) to
address it. First, a novel adversarial mutual information (MI) loss is proposed
to alternately minimize MI with respect to the target classifier and maximize
MI with respect to the feature extractor. By this min-max manner, the positions
of target prototypes are adaptively modified which makes unlabeled images
clustered more easily such that intra-domain gap can be mitigated. Second, to
assist adversarial MI loss, we utilize a graph convolution network to predict
linkage likelihoods between target data and generate pseudo-labels. It
leverages valuable information in the context of nodes and can achieve more
reliable results. The proposed method is evaluated under two scenarios, i.e.,
domain adaptation across poses and image conditions, and domain adaptation
across faces with different skin tones. Extensive experiments show that AIN
successfully improves cross-domain generalization and offers a new
state-of-the-art on RFW dataset.

本文介绍了一种针对人脸识别中源域和目标域数据不匹配问题的方法 —— 通过提出一种新的对抗信息网络来实现无监督域适应，并在两种情况下得到了验证。