In this paper, we investigate self-supervised pre-training methods for
document text recognition. Nowadays, large unlabeled datasets can be collected
for many research tasks, including text recognition, but it is costly to
annotate them. Therefore, methods utilizing unlabeled data are researched. We
study self-supervised pre-training methods based on masked label prediction
using three different approaches -- Feature Quantization, VQ-VAE, and
Post-Quantized AE. We also investigate joint-embedding approaches with VICReg
and NT-Xent objectives, for which we propose an image shifting technique to
prevent model collapse where it relies solely on positional encoding while
completely ignoring the input image. We perform our experiments on historical
handwritten (Bentham) and historical printed datasets mainly to investigate the
benefits of the self-supervised pre-training techniques with different amounts
of annotated target domain data. We use transfer learning as strong baselines.
The evaluation shows that the self-supervised pre-training on data from the
target domain is very effective, but it struggles to outperform transfer
learning from closely related domains. This paper is one of the first
researches exploring self-supervised pre-training in document text recognition,
and we believe that it will become a cornerstone for future research in this
area. We made our implementation of the investigated methods publicly available
at this https URL

本文研究了文档文本识别的自监督预训练方法，探索了利用未标记数据的方法，并提出了基于掩码标签预测的自监督预训练方法，以及联合嵌入方法和模型崩溃预防技术。通过在历史手写和历史印刷数据集上的实验表明，自监督预训练在目标领域数据上非常有效，但在与之密切相关的领域上的迁移学习上却表现不佳。