Multimodal learning is susceptible to modality missing, which poses a major
obstacle for its practical applications and, thus, invigorates increasing
research interest. In this paper, we investigate two challenging problems: 1)
when modality missing exists in the training data, how to exploit the
incomplete samples while guaranteeing that they are properly supervised? 2)
when the missing rates of different modalities vary, causing or exacerbating
the imbalance among modalities, how to address the imbalance and ensure all
modalities are well-trained? To tackle these two challenges, we first introduce
the variational information bottleneck (VIB) method for the cross-modal
representation learning of missing modalities, which capitalizes on the
available modalities and the labels as supervision. Then, accounting for the
imbalanced missing rates, we define relative advantage to quantify the
advantage of each modality over others. Accordingly, a bi-level optimization
problem is formulated to adaptively regulate the supervision of all modalities
during training. As a whole, the proposed approach features \textbf{Re}lative
a\textbf{d}vantage aware \textbf{C}ross-m\textbf{o}dal \textbf{r}epresentation
l\textbf{e}arning (abbreviated as \textbf{RedCore}) for missing modalities with
imbalanced missing rates. Extensive empirical results demonstrate that RedCore
outperforms competing models in that it exhibits superior robustness against
either large or imbalanced missing rates.

我们提出了一种名为 RedCore 的相对优势感知跨模态表示学习方法，用于处理存在缺失模态和具有不平衡缺失率的情况，通过 VIB 方法进行跨模态表示学习并通过双层优化问题自适应调节所有模态的监督，实验证明 RedCore 模型在对大量或不平衡缺失率具有优越的稳健性。