When trying to answer complex questions, people often rely on multiple
sources of information, such as visual, textual, and tabular data. Previous
approaches to this problem have focused on designing input features or model
structure in the multi-modal space, which is inflexible for cross-modal
reasoning or data-efficient training. In this paper, we call for an alternative
paradigm, which transforms the images and tables into unified language
representations, so that we can simplify the task into a simpler textual QA
problem that can be solved using three steps: retrieval, ranking, and
generation, all within a language space. This idea takes advantage of the power
of pre-trained language models and is implemented in a framework called Solar.
Our experimental results show that Solar outperforms all existing methods by
10.6-32.3 pts on two datasets, MultimodalQA and MMCoQA, across ten different
metrics. Additionally, Solar achieves the best performance on the WebQA
leaderboard

本文提出了一种新的跨模态推理方式，即将图像和表格转换为统一的语言表示，以简化任务并利用先前训练好的语言模型，同时介绍了一种名为 Solar 的框架，实验结果表明，Solar 在两个数据集上的表现优于所有现有方法，并在 WebQA 排行榜上取得了最佳成绩。