Multi-modal models have shown a promising capability to effectively integrate
information from various sources, yet meanwhile, they are found vulnerable to
pervasive perturbations, such as uni-modal attacks and missing conditions. To
counter these perturbations, robust multi-modal representations are highly
expected, which are positioned well away from the discriminative multi-modal
decision boundary. In this paper, different from conventional empirical
studies, we focus on a commonly used joint multi-modal framework and
theoretically discover that larger uni-modal representation margins and more
reliable integration for modalities are essential components for achieving
higher robustness. This discovery can further explain the limitation of
multi-modal robustness and the phenomenon that multi-modal models are often
vulnerable to attacks on the specific modality. Moreover, our analysis reveals
how the widespread issue, that the model has different preferences for
modalities, limits the multi-modal robustness by influencing the essential
components and could lead to attacks on the specific modality highly effective.
Inspired by our theoretical finding, we introduce a training procedure called
Certifiable Robust Multi-modal Training (CRMT), which can alleviate this
influence from modality preference and explicitly regulate essential components
to significantly improve robustness in a certifiable manner. Our method
demonstrates substantial improvements in performance and robustness compared
with existing methods. Furthermore, our training procedure can be easily
extended to enhance other robust training strategies, highlighting its
credibility and flexibility.

多模态模型容易受到单模态攻击和缺失条件的干扰，为了克服这些干扰，需要具备鲁棒的多模态表示。本文通过理论研究发现，较大的单模态表示边界和更可靠的模态融合是实现更高鲁棒性的重要组成部分。基于这一发现，引入了一种训练过程称为可证明鲁棒的多模态训练 (Certifiable Robust Multi-modal Training，CRMT)，该方法可以明显提高鲁棒性的可信度和灵活性。