In this paper, we explore the potential of the Contrastive Language-Image
Pretraining (CLIP) model in scene text recognition (STR), and establish a novel
Symmetrical Linguistic Feature Distillation framework (named CLIP-OCR) to
leverage both visual and linguistic knowledge in CLIP. Different from previous
CLIP-based methods mainly considering feature generalization on visual
encoding, we propose a symmetrical distillation strategy (SDS) that further
captures the linguistic knowledge in the CLIP text encoder. By cascading the
CLIP image encoder with the reversed CLIP text encoder, a symmetrical structure
is built with an image-to-text feature flow that covers not only visual but
also linguistic information for distillation.Benefiting from the natural
alignment in CLIP, such guidance flow provides a progressive optimization
objective from vision to language, which can supervise the STR feature
forwarding process layer-by-layer.Besides, a new Linguistic Consistency Loss
(LCL) is proposed to enhance the linguistic capability by considering
second-order statistics during the optimization. Overall, CLIP-OCR is the first
to design a smooth transition between image and text for the STR task.Extensive
experiments demonstrate the effectiveness of CLIP-OCR with 93.8% average
accuracy on six popular STR benchmarks.Code will be available at
this https URL

利用对比语言图像预训练（CLIP）模型在场景文本识别（STR）中的潜力，我们建立了一个新的对称语言特征提取框架（称为 CLIP-OCR），以充分利用 CLIP 中的视觉和语言知识。通过将 CLIP 图像编码器与反转的 CLIP 文本编码器级联，我们构建了一个对称结构，其中包括了从图像到文本的特征传递流，覆盖了视觉和语言信息的提取。实验表明，CLIP-OCR 在六个流行的 STR 基准测试上的平均准确率达到 93.8%。