Multimodal Machine Learning has emerged as a prominent research direction
across various applications such as Sentiment Analysis, Emotion Recognition,
Machine Translation, Hate Speech Recognition, and Movie Genre Classification.
This approach has shown promising results by utilizing modern deep learning
architectures. Despite the achievements made, challenges remain in data
representation, alignment techniques, reasoning, generation, and quantification
within multimodal learning. Additionally, assumptions about the dominant role
of textual modality in decision-making have been made. However, limited
investigations have been conducted on the influence of different modalities in
Multimodal Machine Learning systems. This paper aims to address this gap by
studying the impact of each modality on multimodal learning tasks. The research
focuses on verifying presumptions and gaining insights into the usage of
different modalities. The main contribution of this work is the proposal of a
methodology to determine the effect of each modality on several Multimodal
Machine Learning models and datasets from various tasks. Specifically, the
study examines Multimodal Sentiment Analysis, Multimodal Emotion Recognition,
Multimodal Hate Speech Recognition, and Multimodal Disease Detection. The study
objectives include training SOTA MultiModal Machine Learning models with masked
modalities to evaluate their impact on performance. Furthermore, the research
aims to identify the most influential modality or set of modalities for each
task and draw conclusions for diverse multimodal classification tasks. By
undertaking these investigations, this research contributes to a better
understanding of the role of individual modalities in multi-modal learning and
provides valuable insights for future advancements in this field.

本研究旨在探讨每种模态对 Multimodal Machine Learning 模型的影响，并针对不同分类任务的数据集和模型，提出了一种确定每种模态对 Multimodal Machine Learning 模型的影响的方法。研究结果对于理解多模态学习中每个模态的作用，并为该领域的未来发展提供了有价值的见解。