Modeling semantic information is helpful for scene text recognition. In this
work, we propose to model semantic and visual information jointly with a
Visual-Semantic Transformer (VST). The VST first explicitly extracts primary
semantic information from visual feature maps with a transformer module and a
primary visual-semantic alignment module. The semantic information is then
joined with the visual feature maps (viewed as a sequence) to form a pseudo
multi-domain sequence combining visual and semantic information, which is
subsequently fed into an transformer-based interaction module to enable
learning of interactions between visual and semantic features. In this way, the
visual features can be enhanced by the semantic information and vice versus.
The enhanced version of visual features are further decoded by a secondary
visual-semantic alignment module which shares weights with the primary one.
Finally, the decoded visual features and the enhanced semantic features are
jointly processed by the third transformer module obtaining the final text
prediction. Experiments on seven public benchmarks including regular/ irregular
text recognition datasets verifies the effectiveness our proposed model,
reaching state of the art on four of the seven benchmarks.

本文提出了一种 Visual-Semantic Transformer 的模型，通过 Transformer 模块和视觉 - 语义对齐模块从视觉特征映射中提取出主要的语义信息，然后将语义信息和视觉特征序列组合成伪多域序列，进而通过交互模块增强视觉特征和语义信息之间的相互作用，从而实现文本识别。实验结果表明，该模型在包括正常 / 不规则文本识别数据集在内的七个公共基准上达到了最先进水平。