In a wide range of multimodal tasks, contrastive learning has become a
particularly appealing approach since it can successfully learn representations
from abundant unlabeled data with only pairing information (e.g., image-caption
or video-audio pairs). Underpinning these approaches is the assumption of
multi-view redundancy - that shared information between modalities is necessary
and sufficient for downstream tasks. However, in many real-world settings,
task-relevant information is also contained in modality-unique regions:
information that is only present in one modality but still relevant to the
task. How can we learn self-supervised multimodal representations to capture
both shared and unique information relevant to downstream tasks? This paper
proposes FactorCL, a new multimodal representation learning method to go beyond
multi-view redundancy. FactorCL is built from three new contributions: (1)
factorizing task-relevant information into shared and unique representations,
(2) capturing task-relevant information via maximizing MI lower bounds and
removing task-irrelevant information via minimizing MI upper bounds, and (3)
multimodal data augmentations to approximate task relevance without labels. On
large-scale real-world datasets, FactorCL captures both shared and unique
information and achieves state-of-the-art results on six benchmarks.

本文提出了 FactorCL，一种新的多模态表示学习方法，旨在捕获有助于下游任务的共享信息和独特信息，该方法通过将任务相关信息分解为共享和独特表示，并通过最大化 MI 下限来捕获任务相关信息以及通过最小化 MI 上限来消除任务不相关的信息，并使用多模态数据增强来近似任务相关性。实验证明 FactorCL 在六项基准测试上取得了最先进的结果。