Integrating and processing information from various sources or modalities are
critical for obtaining a comprehensive and accurate perception of the real
world. Drawing inspiration from neuroscience, we develop the
Information-Theoretic Hierarchical Perception (ITHP) model, which utilizes the
concept of information bottleneck. Distinct from most traditional fusion models
that aim to incorporate all modalities as input, our model designates the prime
modality as input, while the remaining modalities act as detectors in the
information pathway. Our proposed perception model focuses on constructing an
effective and compact information flow by achieving a balance between the
minimization of mutual information between the latent state and the input modal
state, and the maximization of mutual information between the latent states and
the remaining modal states. This approach leads to compact latent state
representations that retain relevant information while minimizing redundancy,
thereby substantially enhancing the performance of downstream tasks.
Experimental evaluations on both the MUStARD and CMU-MOSI datasets demonstrate
that our model consistently distills crucial information in multimodal learning
scenarios, outperforming state-of-the-art benchmarks.

通过信息融合和处理各种来源或方式的信息对于获取综合准确的现实世界感知至关重要。我们从神经科学中获得灵感，开发了基于信息瓶颈概念的信息论层次感知（ITHP）模型。与大多数旨在将所有模态作为输入的传统融合模型不同，我们的模型将主模态定为输入，而其余模态则充当信息传递途径中的探测器。我们的感知模型专注于构建有效且紧凑的信息流，通过在潜变量状态与输入模态状态之间最小化互信息并在潜变量状态与其余模态状态之间最大化互信息之间取得平衡。这种方法导致紧凑的潜变量表示，保留相关信息并尽量减少冗余，从而显著提高了下游任务的性能。对 MUStARD 和 CMU-MOSI 数据集的实验评估表明，我们的模型在多模态学习场景中持续提炼关键信息，优于业界最先进的基准。