Multi-modal methods establish comprehensive superiority over uni-modal
methods. However, the imbalanced contributions of different modalities to
task-dependent predictions constantly degrade the discriminative performance of
canonical multi-modal methods. Based on the contribution to task-dependent
predictions, modalities can be identified as predominant and auxiliary
modalities. Benchmark methods raise a tractable solution: augmenting the
auxiliary modality with a minor contribution during training. However, our
empirical explorations challenge the fundamental idea behind such behavior, and
we further conclude that benchmark approaches suffer from certain defects:
insufficient theoretical interpretability and limited exploration capability of
discriminative knowledge. To this end, we revisit multi-modal representation
learning from a causal perspective and build the Structural Causal Model.
Following the empirical explorations, we determine to capture the true
causality between the discriminative knowledge of predominant modality and
predictive label while considering the auxiliary modality. Thus, we introduce
the $\beta$-generalization front-door criterion. Furthermore, we propose a
novel network for sufficiently exploring multi-modal discriminative knowledge.
Rigorous theoretical analyses and various empirical evaluations are provided to
support the effectiveness of the innate mechanism behind our proposed method.

从因果关系的角度重新审视多模态表示学习，构建结构因果模型，引入 β- 泛化前门准则，通过提出的方法充分探索多模态判别性知识的内在机制的严格理论分析和多种经验评估支持其有效性。

通过 $β$- 广义前门准则进行干预性非平衡多模态表示学习

Interventional Imbalanced Multi-Modal Representation Learning via  $β$-Generalization Front-Door Criterion

Most recent self-supervised learning~(SSL) methods are pre-trained on the
well-curated ImageNet-1K dataset. In this work, we consider SSL pre-training on
noisy web image-text paired data due to the excellent scalability of web data.
First, we conduct a benchmark study of representative SSL pre-training methods
on large-scale web data in a fair condition. Methods include single-modal ones
such as MAE and multi-modal ones such as CLIP. We observe that multi-modal
methods cannot outperform single-modal ones on vision transfer learning tasks.
We derive an information-theoretical view to explain the benchmarking results,
which provides insights into designing novel vision learners. Inspired by the
above explorations, we present a visual representation pre-training method,
MUlti-modal Generator~(MUG), for scalable web image-text data. MUG achieves
state-of-the-art transferring performances on a variety of tasks and shows
promising scaling behavior. Models and codes will be made public. Demo
available at this https URL

本论文研究了基于网络图像 - 文本对数据进行自监督学习的方法，对比单模式和多模式方法在视觉转移学习任务上的表现，提出了一个可扩展的 MUlti-modal Generator 预训练方法，并在多项任务中取得了最先进的转移效果。

视觉学习者遇到的网络图像 - 文本对

Vision Learners Meet Web Image-Text Pairs

Face Anti-spoofing gains increased attentions recently in both academic and
industrial fields. With the emergence of various CNN based solutions, the
multi-modal(RGB, depth and IR) methods based CNN showed better performance than
single modal classifiers. However, there is a need for improving the
performance and reducing the complexity. Therefore, an extreme light network
architecture(FeatherNet A/B) is proposed with a streaming module which fixes
the weakness of Global Average Pooling and uses less parameters. Our single
FeatherNet trained by depth image only, provides a higher baseline with 0.00168
ACER, 0.35M parameters and 83M FLOPS. Furthermore, a novel fusion procedure
with ``ensemble + cascade'' structure is presented to satisfy the performance
preferred use cases. Meanwhile, the MMFD dataset is collected to provide more
attacks and diversity to gain better generalization. We use the fusion method
in the Face Anti-spoofing Attack Detection Challenge@CVPR2019 and got the
result of 0.0013(ACER), 0.999(TPR@FPR=10e-2), 0.998(TPR@FPR=10e-3) and
0.9814(TPR@FPR=10e-4).

本研究提出了极致轻量化的神经网络结构，同时设计了一种新颖的集成融合方法和数据集，以提高人脸反欺诈检测的性能和鲁棒性。