With the continuous development of OCR technology and the expansion of
application fields, text recognition in complex scenes has become a key
challenge. Factors such as multiple fonts, mixed scenes and complex layouts
seriously affect the recognition accuracy of traditional OCR models. Although
OCR models based on deep learning have performed well in specific fields or
similar data sets in recent years, the generalization ability and robustness of
the model are still a big challenge when facing complex environments with
multiple scenes. Furthermore, training an OCR model from scratch or fine-tuning
all parameters is very demanding on computing resources and inference time,
which limits the flexibility of its application. This study focuses on a
fundamental aspect of mixed text recognition in response to the challenges
mentioned above, which involves effectively fine-tuning the pre-trained basic
OCR model to demonstrate exceptional performance across various downstream
tasks. To this end, we propose a parameter-efficient hybrid text recognition
method based on pre-trained OCR Transformer, namely DLoRA-TrOCR. This method
embeds DoRA into the image encoder and LoRA into the internal structure of the
text decoder, enabling efficient parameter fine-tuning for downstream tasks.
Experimental results show that compared to similar parameter adjustment
methods, our model DLoRA-TrOCR has the smallest number of parameters and
performs better. It can achieve state-of-the-art performance on complex scene
data sets involving simultaneous recognition of mixed handwritten, printed and
street view texts.

本研究旨在通过对预训练基础 OCR 模型进行有效的参数微调，在各种下游任务中展示出卓越的性能。我们提出了一种基于预训练 OCR Transformer 的参数高效混合文本识别方法，即 DLoRA-TrOCR。该方法将 DoRA 嵌入图像编码器和 LoRA 嵌入文本解码器的内部结构，使得下游任务的参数微调更加高效。实验结果表明，与类似的参数调整方法相比，我们的模型 DLoRA-TrOCR 具有最少的参数，并且表现更好。它可以在包括混合手写、印刷和街景文本的复杂场景数据集上取得最先进的性能。