Many visual scenes contain text that carries crucial information, and it is
thus essential to understand text in images for downstream reasoning tasks. For
example, a deep water label on a warning sign warns people about the danger in
the scene. Recent work has explored the TextVQA task that requires reading and
understanding text in images to answer a question. However, existing approaches
for TextVQA are mostly based on custom pairwise fusion mechanisms between a
pair of two modalities and are restricted to a single prediction step by
casting TextVQA as a classification task. In this work, we propose a novel
model for the TextVQA task based on a multimodal transformer architecture
accompanied by a rich representation for text in images. Our model naturally
fuses different modalities homogeneously by embedding them into a common
semantic space where self-attention is applied to model inter- and intra-
modality context. Furthermore, it enables iterative answer decoding with a
dynamic pointer network, allowing the model to form an answer through
multi-step prediction instead of one-step classification. Our model outperforms
existing approaches on three benchmark datasets for the TextVQA task by a large
margin.

本研究提出了一种基于多模态 Transformer 架构的新型 TextVQA 模型，采用自我注意力机制进行不同模态的融合，并通过动态指针网络进行迭代式答案解码，相比于现有方法大幅度提高了 TextVQA 任务的性能。