Scene text recognition, as a cross-modal task involving vision and text, is
an important research topic in computer vision. Most existing methods use
language models to extract semantic information for optimizing visual
recognition. However, the guidance of visual cues is ignored in the process of
semantic mining, which limits the performance of the algorithm in recognizing
irregular scene text. To tackle this issue, we propose a novel cross-modal
fusion network (CMFN) for irregular scene text recognition, which incorporates
visual cues into the semantic mining process. Specifically, CMFN consists of a
position self-enhanced encoder, a visual recognition branch and an iterative
semantic recognition branch. The position self-enhanced encoder provides
character sequence position encoding for both the visual recognition branch and
the iterative semantic recognition branch. The visual recognition branch
carries out visual recognition based on the visual features extracted by CNN
and the position encoding information provided by the position self-enhanced
encoder. The iterative semantic recognition branch, which consists of a
language recognition module and a cross-modal fusion gate, simulates the way
that human recognizes scene text and integrates cross-modal visual cues for
text recognition. The experiments demonstrate that the proposed CMFN algorithm
achieves comparable performance to state-of-the-art algorithms, indicating its
effectiveness.

提出了一种新颖的交叉模态融合网络（CMFN），用于不规则场景文本识别，将视觉线索融入语义挖掘过程中，实验证明该算法在性能上与最先进的算法相当，验证了其有效性。

CMFN: 跨模态融合网络用于不规则场景文字识别

CMFN: Cross-Modal Fusion Network for Irregular Scene Text Recognition

Existing text recognition methods usually need large-scale training data.
Most of them rely on synthetic training data due to the lack of annotated real
images. However, there is a domain gap between the synthetic data and real
data, which limits the performance of the text recognition models. Recent
self-supervised text recognition methods attempted to utilize unlabeled real
images by introducing contrastive learning, which mainly learns the
discrimination of the text images. Inspired by the observation that humans
learn to recognize the texts through both reading and writing, we propose to
learn discrimination and generation by integrating contrastive learning and
masked image modeling in our self-supervised method. The contrastive learning
branch is adopted to learn the discrimination of text images, which imitates
the reading behavior of humans. Meanwhile, masked image modeling is firstly
introduced for text recognition to learn the context generation of the text
images, which is similar to the writing behavior. The experimental results show
that our method outperforms previous self-supervised text recognition methods
by 10.2%-20.2% on irregular scene text recognition datasets. Moreover, our
proposed text recognizer exceeds previous state-of-the-art text recognition
methods by averagely 5.3% on 11 benchmarks, with similar model size. We also
demonstrate that our pre-trained model can be easily applied to other
text-related tasks with obvious performance gain. The code is available at
this https URL

本文提出一种基于对比学习和遮挡图像建模的自监督文本识别方法，实验结果显示，该方法在不规则场景文本识别数据集上优于之前的自监督文本识别方法 10.2％-20.2％，同时，在 11 个基准测试中，该文本识别器的表现也超过了之前的最新方法平均 5.3％，且能够轻松适用于其他文本相关任务。