Existing studies on multimodal sentiment analysis heavily rely on textual
modality and unavoidably induce the spurious correlations between textual words
and sentiment labels. This greatly hinders the model generalization ability. To
address this problem, we define the task of out-of-distribution (OOD)
multimodal sentiment analysis. This task aims to estimate and mitigate the bad
effect of textual modality for strong OOD generalization. To this end, we
embrace causal inference, which inspects the causal relationships via a causal
graph. From the graph, we find that the spurious correlations are attributed to
the direct effect of textual modality on the model prediction while the
indirect one is more reliable by considering multimodal semantics. Inspired by
this, we devise a model-agnostic counterfactual framework for multimodal
sentiment analysis, which captures the direct effect of textual modality via an
extra text model and estimates the indirect one by a multimodal model. During
the inference, we first estimate the direct effect by the counterfactual
inference, and then subtract it from the total effect of all modalities to
obtain the indirect effect for reliable prediction. Extensive experiments show
the superior effectiveness and generalization ability of our proposed
framework.

本文提出了一种基于因果推断的针对 out-of-distribution (OOD) 多模态情感分析的方法，通过引入反事实框架以及考虑多模态语义，分别考虑文本的直接和间接效应，对多模态情感进行可靠的预测并展示了该方法在泛化能力方面的优越性。

基于反事实推理的跨领域多模态情感分析

Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis

We present a novel counterfactual framework for both Zero-Shot Learning (ZSL)
and Open-Set Recognition (OSR), whose common challenge is generalizing to the
unseen-classes by only training on the seen-classes. Our idea stems from the
observation that the generated samples for unseen-classes are often out of the
true distribution, which causes severe recognition rate imbalance between the
seen-class (high) and unseen-class (low). We show that the key reason is that
the generation is not Counterfactual Faithful, and thus we propose a faithful
one, whose generation is from the sample-specific counterfactual question: What
would the sample look like, if we set its class attribute to a certain class,
while keeping its sample attribute unchanged? Thanks to the faithfulness, we
can apply the Consistency Rule to perform unseen/seen binary classification, by
asking: Would its counterfactual still look like itself? If ``yes'', the sample
is from a certain class, and ``no'' otherwise. Through extensive experiments on
ZSL and OSR, we demonstrate that our framework effectively mitigates the
seen/unseen imbalance and hence significantly improves the overall performance.
Note that this framework is orthogonal to existing methods, thus, it can serve
as a new baseline to evaluate how ZSL/OSR models generalize. Codes are
available at this https URL

提出了一种针对零样本学习和开放式识别的可靠反事实框架，通过引入计数事实的概念来对未知类别进行样本生成，使用一致性规则进行二元分类，从而有效地减轻了样本类别分布不平衡的问题。

反事实零样本和开放集视觉识别

Counterfactual Zero-Shot and Open-Set Visual Recognition

Questions concerning mediated causal effects are of great interest in
psychology, cognitive science, medicine, social science, public health, and
many other disciplines. For instance, about 60% of recent papers published in
leading journals in social psychology contain at least one mediation test
(Rucker, Preacher, Tormala, & Petty, 2011). Standard parametric approaches to
mediation analysis employ regression models, and either the "difference method"
(Judd & Kenny, 1981), more common in epidemiology, or the "product method"
(Baron & Kenny, 1986), more common in the social sciences. In this paper we
first discuss a known, but perhaps often unappreciated fact: that these
parametric approaches are a special case of a general counterfactual framework
for reasoning about causality first described by Neyman (1923), and Rubin
(1974), and linked to causal graphical models by J. Robins (1986), and Pearl
(2000). We then show a number of advantages of this framework. First, it makes
the strong assumptions underlying mediation analysis explicit. Second, it
avoids a number of problems present in the product and difference methods, such
as biased estimates of effects in certain cases. Finally, we show the
generality of this framework by proving a novel result which allows mediation
analysis to be applied to longitudinal settings with unobserved confounders.

该研究论文讨论了介导分析的因果效应、参数方法以及对反事实框架的一般性应用。