Multi-modal reasoning plays a vital role in bridging the gap between textual
and visual information, enabling a deeper understanding of the context. This
paper presents the Feature Swapping Multi-modal Reasoning (FSMR) model,
designed to enhance multi-modal reasoning through feature swapping. FSMR
leverages a pre-trained visual-language model as an encoder, accommodating both
text and image inputs for effective feature representation from both
modalities. It introduces a unique feature swapping module, enabling the
exchange of features between identified objects in images and corresponding
vocabulary words in text, thereby enhancing the model's comprehension of the
interplay between images and text. To further bolster its multi-modal alignment
capabilities, FSMR incorporates a multi-modal cross-attention mechanism,
facilitating the joint modeling of textual and visual information. During
training, we employ image-text matching and cross-entropy losses to ensure
semantic consistency between visual and language elements. Extensive
experiments on the PMR dataset demonstrate FSMR's superiority over
state-of-the-art baseline models across various performance metrics.

该论文介绍了一种名为特征交换多模态推理（FSMR）模型的方法，通过特征交换来增强多模态推理，提供了图像和文本的有效特征表示，并加强了模型对图像和文本之间相互作用的理解，通过多模态交叉注意机制实现了文本和视觉信息的联合建模，实验证明 FSMR 在各种性能指标上优于基准模型。