Multimodal contrastive representation learning methods have proven successful
across a range of domains, partly due to their ability to generate meaningful
shared representations of complex phenomena. To enhance the depth of analysis
and understanding of these acquired representations, we introduce a unified
causal model specifically designed for multimodal data. By examining this
model, we show that multimodal contrastive representation learning excels at
identifying latent coupled variables within the proposed unified model, up to
linear or permutation transformations resulting from different assumptions. Our
findings illuminate the potential of pre-trained multimodal models, eg, CLIP,
in learning disentangled representations through a surprisingly simple yet
highly effective tool: linear independent component analysis. Experiments
demonstrate the robustness of our findings, even when the assumptions are
violated, and validate the effectiveness of the proposed method in learning
disentangled representations.

用于多模态数据的统一因果模型，通过线性独立成分分析的方法，可以提高对多模态对比表示学习的理解和分析深度，发现预训练的多模态模型可以通过学习解缠表示来提高其性能和鲁棒性。

通过潜在的部分因果模型揭示多模态对比表示学习

Revealing Multimodal Contrastive Representation Learning through Latent  Partial Causal Models

Pre-trained multimodal models have achieved significant success in
retrieval-based question answering. However, current multimodal retrieval
question-answering models face two main challenges. Firstly, utilizing
compressed evidence features as input to the model results in the loss of
fine-grained information within the evidence. Secondly, a gap exists between
the feature extraction of evidence and the question, which hinders the model
from effectively extracting critical features from the evidence based on the
given question. We propose a two-stage framework for evidence retrieval and
question-answering to alleviate these issues. First and foremost, we propose a
progressive evidence refinement strategy for selecting crucial evidence. This
strategy employs an iterative evidence retrieval approach to uncover the
logical sequence among the evidence pieces. It incorporates two rounds of
filtering to optimize the solution space, thus further ensuring temporal
efficiency. Subsequently, we introduce a semi-supervised contrastive learning
training strategy based on negative samples to expand the scope of the question
domain, allowing for a more thorough exploration of latent knowledge within
known samples. Finally, in order to mitigate the loss of fine-grained
information, we devise a multi-turn retrieval and question-answering strategy
to handle multimodal inputs. This strategy involves incorporating multimodal
evidence directly into the model as part of the historical dialogue and
question. Meanwhile, we leverage a cross-modal attention mechanism to capture
the underlying connections between the evidence and the question, and the
answer is generated through a decoding generation approach. We validate the
model's effectiveness through extensive experiments, achieving outstanding
performance on WebQA and MultimodelQA benchmark tests.

我们提出了一个基于预训练多模态模型的证据检索和问答的两阶段框架来解决模型面临的两个主要挑战：利用压缩的证据特征会导致丢失细粒度信息以及提取问题关键特征时存在特征提取和问题之间的差距。我们的方法包括渐进证据精炼策略来选择关键证据、半监督对比学习训练策略扩展问题领域范围以及多轮检索和问答策略来处理多模态输入。通过大量实验证明了模型的有效性，在 WebQA 和 MultimodelQA 基准测试上取得了出色的性能。

开放域多模式检索问答的逐步证据细化

Progressive Evidence Refinement for Open-domain Multimodal Retrieval  Question Answering

Large pre-trained multimodal models have demonstrated significant success in
a range of downstream tasks, including image captioning, image-text retrieval,
visual question answering (VQA), etc. However, many of these methods rely on
image-text pairs collected from the web as pre-training data and unfortunately
overlook the need for fine-grained feature alignment between vision and
language modalities, which requires detailed understanding of images and
language expressions. While integrating VQA and dense captioning (DC) into
pre-training can address this issue, acquiring image-question-answer as well as
image-location-caption triplets is challenging and time-consuming.
Additionally, publicly available datasets for VQA and dense captioning are
typically limited in scale due to manual data collection and labeling efforts.
In this paper, we propose a novel method called Joint QA and DC GEneration
(JADE), which utilizes a pre-trained multimodal model and easily-crawled
image-text pairs to automatically generate and filter large-scale VQA and dense
captioning datasets. We apply this method to the Conceptual Caption (CC3M)
dataset to generate a new dataset called CC3M-QA-DC. Experiments show that when
used for pre-training in a multi-task manner, CC3M-QA-DC can improve the
performance with various backbones on various downstream tasks. Furthermore,
our generated CC3M-QA-DC can be combined with larger image-text datasets (e.g.,
CC15M) and achieve competitive results compared with models using much more
data. Code and dataset will be released.

本文提出一种名为 Joint QA and DC Generation (JADE) 的新方法，利用预训练的多模态模型及易于爬取的图像 - 文本对生成和过滤大规模的视觉问答和密集字幕数据集。我们将该方法应用于概念字幕（CC3M）数据集，生成了一个名为 CC3M-QA-DC 的新的数据集，在多任务方式预训练时，CC3M-QA-DC 可以改善各种骨干网络在各种下游任务中的性能，并与更多数据使用模型相比，我们生成的 CC3M-QA-DC 和更大的图像 - 文本数据集（例如 CC15M）相结合，在相同的计算条件下达到了有竞争力的结果。