Scene text removal (STR) aims at replacing text strokes in natural scenes
with visually coherent backgrounds. Recent STR approaches rely on iterative
refinements or explicit text masks, resulting in higher complexity and
sensitivity to the accuracy of text localization. Moreover, most existing STR
methods utilize convolutional neural networks (CNNs) for feature representation
while the potential of vision Transformers (ViTs) remains largely unexplored.
In this paper, we propose a simple-yet-effective ViT-based text eraser, dubbed
ViTEraser. Following a concise encoder-decoder framework, different types of
ViTs can be easily integrated into ViTEraser to enhance the long-range
dependencies and global reasoning. Specifically, the encoder hierarchically
maps the input image into the hidden space through ViT blocks and patch
embedding layers, while the decoder gradually upsamples the hidden features to
the text-erased image with ViT blocks and patch splitting layers. As ViTEraser
implicitly integrates text localization and inpainting, we propose a novel
end-to-end pretraining method, termed SegMIM, which focuses the encoder and
decoder on the text box segmentation and masked image modeling tasks,
respectively. To verify the effectiveness of the proposed methods, we
comprehensively explore the architecture, pretraining, and scalability of the
ViT-based encoder-decoder for STR, which provides deep insights into the
application of ViT to STR. Experimental results demonstrate that ViTEraser with
SegMIM achieves state-of-the-art performance on STR by a substantial margin.
Furthermore, the extended experiment on tampered scene text detection
demonstrates the generality of ViTEraser to other tasks. We believe this paper
can inspire more research on ViT-based STR approaches. Code will be available
at this https URL

本研究提出了一种基于 ViTs 的简单而有效的文本擦除器，在编码器 - 解码器框架内集成文本定位和修复，通过 SegMIM 方法进行端到端预训练，实现了在场景文本去除方面的最先进性能提升，表明了 ViTs 在此领域的广泛应用前景。