Integrating and processing information from various sources or modalities are
critical for obtaining a comprehensive and accurate perception of the real
world in autonomous systems and cyber-physical systems. Drawing inspiration
from neuroscience, we develop the Information-Theoretic Hierarchical Perception
(ITHP) model, which utilizes the concept of information bottleneck. Different
from most traditional fusion models that incorporate all modalities identically
in neural networks, our model designates a prime modality and regards the
remaining modalities as detectors in the information pathway, serving to
distill the flow of information. Our proposed perception model focuses on
constructing an effective and compact information flow by achieving a balance
between the minimization of mutual information between the latent state and the
input modal state, and the maximization of mutual information between the
latent states and the remaining modal states. This approach leads to compact
latent state representations that retain relevant information while minimizing
redundancy, thereby substantially enhancing the performance of multimodal
representation learning. Experimental evaluations on the MUStARD, CMU-MOSI, and
CMU-MOSEI datasets demonstrate that our model consistently distills crucial
information in multimodal learning scenarios, outperforming state-of-the-art
benchmarks. Remarkably, on the CMU-MOSI dataset, ITHP surpasses human-level
performance in the multimodal sentiment binary classification task across all
evaluation metrics (i.e., Binary Accuracy, F1 Score, Mean Absolute Error, and
Pearson Correlation).

通过神经科学的启发，我们开发了信息论层次感知（ITHP）模型，它利用了信息瓶颈的概念。与大多数在神经网络中相同并一致地纳入所有模态的传统融合模型不同，我们的模型将一个主要模态设计为信息路径中的侦测器，以蒸馏信息流。我们提出的感知模型专注于通过在潜在状态与输入模态状态之间最小化互信息，以及在潜在状态与其余模态状态之间最大化互信息来构建一种有效且紧凑的信息流。该方法导致紧凑的潜在状态表示，保留相关信息并最小化冗余，从而大大提高了多模态表示学习的性能。在 MUStARD、CMU-MOSI 和 CMU-MOSEI 数据集上的实验评估表明，我们的模型在多模态学习场景中一贯蒸馏关键信息，优于最先进的基准模型。在 CMU-MOSI 数据集上，ITHP 在多模态情感二分类任务的所有评估指标（二元准确率、F1 分数、平均绝对误差和皮尔逊相关性）上均超过了人类水平的性能。

神经启发的信息理论层次感知多模态学习

Neuro-Inspired Information-Theoretic Hierarchical Perception for  Multimodal Learning

Integrating and processing information from various sources or modalities are
critical for obtaining a comprehensive and accurate perception of the real
world. Drawing inspiration from neuroscience, we develop the
Information-Theoretic Hierarchical Perception (ITHP) model, which utilizes the
concept of information bottleneck. Distinct from most traditional fusion models
that aim to incorporate all modalities as input, our model designates the prime
modality as input, while the remaining modalities act as detectors in the
information pathway. Our proposed perception model focuses on constructing an
effective and compact information flow by achieving a balance between the
minimization of mutual information between the latent state and the input modal
state, and the maximization of mutual information between the latent states and
the remaining modal states. This approach leads to compact latent state
representations that retain relevant information while minimizing redundancy,
thereby substantially enhancing the performance of downstream tasks.
Experimental evaluations on both the MUStARD and CMU-MOSI datasets demonstrate
that our model consistently distills crucial information in multimodal learning
scenarios, outperforming state-of-the-art benchmarks.

通过信息融合和处理各种来源或方式的信息对于获取综合准确的现实世界感知至关重要。我们从神经科学中获得灵感，开发了基于信息瓶颈概念的信息论层次感知（ITHP）模型。与大多数旨在将所有模态作为输入的传统融合模型不同，我们的模型将主模态定为输入，而其余模态则充当信息传递途径中的探测器。我们的感知模型专注于构建有效且紧凑的信息流，通过在潜变量状态与输入模态状态之间最小化互信息并在潜变量状态与其余模态状态之间最大化互信息之间取得平衡。这种方法导致紧凑的潜变量表示，保留相关信息并尽量减少冗余，从而显著提高了下游任务的性能。对 MUStARD 和 CMU-MOSI 数据集的实验评估表明，我们的模型在多模态学习场景中持续提炼关键信息，优于业界最先进的基准。