We present new results to model and understand the role of encoder-decoder
design in machine learning (ML) from an information-theoretic angle. We use two
main information concepts, information sufficiency (IS) and mutual information
loss (MIL), to represent predictive structures in machine learning. Our first
main result provides a functional expression that characterizes the class of
probabilistic models consistent with an IS encoder-decoder latent predictive
structure. This result formally justifies the encoder-decoder forward stages
many modern ML architectures adopt to learn latent (compressed) representations
for classification. To illustrate IS as a realistic and relevant model
assumption, we revisit some known ML concepts and present some interesting new
examples: invariant, robust, sparse, and digital models. Furthermore, our IS
characterization allows us to tackle the fundamental question of how much
performance (predictive expressiveness) could be lost, using the cross entropy
risk, when a given encoder-decoder architecture is adopted in a learning
setting. Here, our second main result shows that a mutual information loss
quantifies the lack of expressiveness attributed to the choice of a (biased)
encoder-decoder ML design. Finally, we address the problem of universal
cross-entropy learning with an encoder-decoder design where necessary and
sufficiency conditions are established to meet this requirement. In all these
results, Shannon's information measures offer new interpretations and
explanations for representation learning.

我们从信息论的角度提出了有关编码器 - 解码器设计在机器学习中发挥作用的新结果，使用信息充分性和互信息损失来表示机器学习中的预测结构，并使用香农的信息度量为表示学习提供了新的解释。

机器学习中编码解码结构的信息度量理解

Understanding Encoder-Decoder Structures in Machine Learning Using  Information Measures

Multi-phase computed tomography (CT) scans use contrast agents to highlight
different anatomical structures within the body to improve the probability of
identifying and detecting anatomical structures of interest and abnormalities
such as liver lesions. Yet, detecting these lesions remains a challenging task
as these lesions vary significantly in their size, shape, texture, and contrast
with respect to surrounding tissue. Therefore, radiologists need to have an
extensive experience to be able to identify and detect these lesions.
Segmentation-based neural networks can assist radiologists with this task.
Current state-of-the-art lesion segmentation networks use the encoder-decoder
design paradigm based on the UNet architecture where the multi-phase CT scan
volume is fed to the network as a multi-channel input. Although this approach
utilizes information from all the phases and outperform single-phase
segmentation networks, we demonstrate that their performance is not optimal and
can be further improved by incorporating the learning from models trained on
each single-phase individually. Our approach comprises three stages. The first
stage identifies the regions within the liver where there might be lesions at
three different scales (4, 8, and 16 mm). The second stage includes the main
segmentation model trained using all the phases as well as a segmentation model
trained on each of the phases individually. The third stage uses the
multi-phase CT volumes together with the predictions from each of the
segmentation models to generate the final segmentation map. Overall, our
approach improves relative liver lesion segmentation performance by 1.6% while
reducing performance variability across subjects by 8% when compared to the
current state-of-the-art models.

通过在多个阶段运用神经网络进行多相计算机断层扫描，结合分割模型和单相训练，我们的方法在肝脏病变分割性能上取得了 1.6% 的改进，并降低了 8% 的主体间性能差异。