Multimodal Variational Autoencoders (VAEs) represent a promising group of
generative models that facilitate the construction of a tractable posterior
within the latent space, given multiple modalities. Daunhawer et al. (2022)
demonstrate that as the number of modalities increases, the generative quality
of each modality declines. In this study, we explore an alternative approach to
enhance the generative performance of multimodal VAEs by jointly modeling the
latent space of unimodal VAEs using score-based models (SBMs). The role of the
SBM is to enforce multimodal coherence by learning the correlation among the
latent variables. Consequently, our model combines the superior generative
quality of unimodal VAEs with coherent integration across different modalities.

本研究提出一种新的方法，通过使用基于分数的模型（SBMs）共同建模单模 VAEs 的潜在空间，以增强多模态 VAEs 的生成性能。该模型利用 SBM 学习潜在变量之间的相关性来强制执行多模态一致性，从而将单模 VAEs 的优异生成质量与不同模态的协同集成相结合。

基于得分的多模态自编码器

Score-Based Multimodal Autoencoders

In the field of multimodal sentiment analysis (MSA), a few studies have
leveraged the inherent modality correlation information stored in samples for
self-supervised learning. However, they feed the training pairs in a random
order without consideration of difficulty. Without human annotation, the
generated training pairs of self-supervised learning often contain noise. If
noisy or hard pairs are used for training at the easy stage, the model might be
stuck in bad local optimum. In this paper, we inject curriculum learning into
weakly supervised modality correlation learning. The weakly supervised
correlation learning leverages the label information to generate scores for
negative pairs to learn a more discriminative embedding space, where negative
pairs are defined as two unimodal embeddings from different samples. To assist
the correlation learning, we feed the training pairs to the model according to
difficulty by the proposed curriculum learning, which consists of elaborately
designed scoring and feeding functions. The scoring function computes the
difficulty of pairs using pre-trained and current correlation predictors, where
the pairs with large losses are defined as hard pairs. Notably, the hardest
pairs are discarded in our algorithm, which are assumed as noisy pairs.
Moreover, the feeding function takes the difference of correlation losses as
feedback to determine the feeding actions (`stay', `step back', or `step
forward'). The proposed method reaches state-of-the-art performance on MSA.

本文将课程学习引入到弱监督的模态相关学习中，结合特定的评分和馈送功能，在多模情感分析任务中达到最优效果。